MULTI-LEVEL EXAMINATION OF LEADERS AND THERAPIST ATTITUDES TOWARDS EVIDENCE-BASED PRACTICES WITHIN A LARGE SCALE SYSTEM REFORM

Transcription

1 MULTI-LEVEL EXAMINATION OF LEADERS AND THERAPIST ATTITUDES TOWARDS EVIDENCE-BASED PRACTICES WITHIN A LARGE SCALE SYSTEM REFORM Nicole Stadnick, Miya Barnett, Jennifer Regan, Anna Lau, & Lauren Brookman-Frazee Implementation Science Seminar April 21, 2016

2 Objectives 1. To understand the multi-level landscape of a local large-scale mental health system reformation. 2. Identify practice, therapist, and organizational characteristics that impact multi-informant attitudes towards practices. 3. To recognize differences in therapist versus leader perspectives on the implementation of multiple evidence-based practices.

3 Factors Affecting Implementation & Sustainment of Evidence-Based Practice (EBP) Service System Agencies Therapists Intervention Adapted from Aarons, Hurlburt & Horwitz, 2011

4 Study Context Los Angeles County: nation s largest county mental health department 75 directly operated sites 288 contracted agencies (120 with child MH services) 8 Service Provision Areas range from rural to inner city communities Serves an ethnically diverse, disadvantaged population Hispanic/Latino Asian/Pacific Islander African American Non-Hispanic White Native American

5 The Prevention and Early Intervention (PEI) Transformation of Los Angeles County Launched August 2009 PEI funds reimburse delivery of specific practices. Contracts with agencies were amended, therapists could bill to one of 52 approved interventions 6 practices initially selected for supported implementation, therapist trainings commenced in April ,100 clients were served by 8,514 MH therapists who delivered at least 1 of the initial 6 practices between FY

6 What is 4KEEPS? o o o Knowledge Exchange on Evidence- Based Practice Sustainment A neutral, observational study investigating the sustainment of original practices with implementation support Data extracted from the ongoing 4KEEPS Study Funded by NIMH Grant # R01 MH100134

7

8 4KEEPS Study Aims Aim 1: Characterize sustainment outcomes Aim 2: Use mixed methods to characterize inner context factors and early implementation conditions that potentially predict EBP sustainment. Aim 3: Identify inner context and early implementation conditions that determine sustainment outcomes Funded by NIMH Grant # R01 MH100134

9 Spotlight on Therapist and Leader Attitudes Positive attitudes towards EBP may impact the willingness of the therapist to: Adopt a practice Implement the practice with fidelity Sustain use of the practice Remain in their current position Minimal quantitative research on leader attitudes towards EBPs Qualitative studies suggest that leaders are more concerned with research and outcomes supporting an EBP, beneficial features of EBP, and costs.

10 Methods Identified agencies contracted to deliver at least 1 of 6 original practices Contacted leaders and district chiefs to identify eligible therapists and leaders and obtain contact information Offered opt-in option to have leaders web-based survey to staff for those agencies who did not want to distribute contact information Individualized link to survey sent through Qualtrics Participants received $20 for completing

11 Characteristics of Practices Practice Age Range (years) Target Problem(s) Triple P 2-12 conduct CPP 0-6 trauma; attachment MAP 0-21 anxiety; trauma; depression; conduct TF-CBT 3-18 trauma CBITS trauma Seeking Safety trauma; substance use

12 How did we measure attitudes? Practice-Specific Attitudes General Attitudes towards EBP Perceived Characteristics of Intervention Scale, Cook et al., 2014 Relative Advantage Complexity Compatibility Potential for Reinvention Total Evidence-Based Practice Attitudes Scale, Aarons, 2004 Openness Divergence

13 Survey 1 Respondents 69 agencies (70% response rate) 173 leaders (executive & program) (60% response rate) 820 therapists (44% response rate)

14 Practice Characteristics Associated with Therapist Attitudes towards Practices Service System Agencies Therapists Intervention

15 Therapist Perceptions of EBP vs. vs. vs.

16 Attitudes Differ by Practice Less favorable More favorable Very Great Extent *** 2.45 ** 3.17 * ** *** *** Not at All CBITS SS MAP Triple P CPP TFCBT CBITS SS MAP Triple P CPP TFCBT *p<.05; **p<.01; ***p<.001

17 Practice Characteristics Practice PCIS Alphas CBITS (N = 65).96 Narrow Age Range Consultation Required Prescribed Session Content/ Order CPP (N = 140).94 MAP (N = 527).92 SS (N = 491).93 TF-CBT (N = 582).93 Triple P (N = 184).94

18 Therapist Characteristics (n = 794) M (SD) or % Gender (Female) 87.9% Licensed 45.2% Race/Ethnicity Non-Hispanic White 33.8% Hispanic 42.5% Other Minority 23.7% Education B.A. and under 1.9% Master s 85.4% Ph.D. 11.8% Caseload (10.46) Hours in Direct Service (8.64)

19 Methods Therapist Practice 1 Practice 2 Practice 3 MIXED Model Level 1: Practice Characteristics Level 2: Therapist Characteristics Gender Race/Ethnicity Licensed Burnout Burden General EBP attitudes

20 Therapist Attitudes Differ by Practice Characteristics Very Great Extent 5 4 PCIS ITEM MEAN 3 * * * Yes No 2 Not at All 1 Prescribed Session Content/ Order Consultation Required Narrow Age Range * = p <.001 PRACTICE CHARACTERISTIC

21 * = p <.05, ** = p <.01, *** = p <.001 Prescribed Session Content/ Order Consultation Required Narrow Age Range F b F b F b Intercept *** 2.88*** *** 2.73*** *** 2.40*** Practice Characteristic 78.58***.35*** 31.44***.24*** 68.87**.35*** Level 2: Therapist Characteristics Gender Licensed 12.30*** -.18*** 10.30**.17** 11.30**.17** Race/Ethnicity Hispanic * --.12* Other Minority Education B.A. and under Ph.D Burnout 34.33*** -.01** 33.83*** -.01*** 34.51*** -.01*** Burden EBPAS Openness 85.78***.34*** 87.92***.35*** 89.22***.35*** EBPAS Divergence

22 Organizational Characteristics Associated with Leader and Therapist Attitudes Service System Agencies Therapist Intervention

23 Leader and Therapist Attitudes Therapists and leaders may value unique aspects of EBPs that impact their attitudes towards them Organizational climate and culture have been shown to impact adoption and implementation of EBPs (Aarons & Sawitzky, 2006; Aarons et al., 2012) Research needed to understand if therapist and leader attitudes differ and the association of organizational characteristics with attitudes Research Questions: 1. Do therapists and leaders differ in their attitudes towards practices? 2. Which organizational characteristics impact therapist and leader attitudes towards practices?

24 Methods Perceived Organizational Characteristics Leaders (n=157) Therapist (n=756) EBP-Specific Attitudes (PCIS) MIXED Model Level 1: Practice EBP-specific attitudes Level 2: Therapist and Leader Perceptions of organizational characteristics Level 3: Agency

25 Predictor Measures: Perceptions of Organizational Characteristics Organizational Climate Measure (Patterson et al., 2005) Autonomy Performance Feedback Involvement Therapists & Leaders Maslach Burnout Inventory (adapted) (Schaufeli & Leiter, 1996) Therapists Organizational Readiness for Change (Lehman, Greener, & Simpson, 2002) Staffing Cohesion Stress Leaders

26 Therapist and Leader Respondents Therapist (n = 756) M (SD) or % Leader (n = 157) M (SD) or % Age (9.08) (9.75) Gender (Female) 88% 84% Primary MH Discipline MFT 55% 53% Psychology 13% 12% Social Work 30% 35% Case Manager 2% 0% Caseload (9.52) (8.42) Administrative Hours (9.31) Clinical Hours (8.44) Years at Agency in Designated Role 6.30 (5.77) 4.59 (4.40)

27 Results: Therapist and Leader Attitudes Very Great Extent 5 EBP-Specific Attitudes by Respondent and Practice Therapist Leader PCIS Item Mean 4 3 *** *** 2 1 Not at All All Practices CBITS CPP MAP SS TFCBT Triple P ***p<.001

28 **p<.01; ***p<.001 Therapist Burnout and Perceived Organizational Climate on Practice Attitudes F Intercept (grand mean) *** 2.65*** Level 1: Practice (Effect Coding) MAP CPP 13.86***.33*** Triple P 15.56***.30*** SS 11.83** -.19** TF-CBT 71.36***.44*** CBITS 42.78*** -.82*** Level 2: Therapist Characteristics Burnout OCM Autonomy OCM Performance Feedback 10.81**.18** OCM Involvement b

29 **p<.01; ***p<.001 Therapist Burnout and Perceived Organizational Climate on Practice Attitudes Burnout OCM Autonomy OCM Performance Feedback b OCM Involvement Intercept 3.31*** 3.30*** 3.30*** 3.30*** Level 1 x 2 MAP CPP.01***.12***.11.12*** Triple P.01***.12***.10.11*** SS -.01*** -.06** *** TF-CBT.01***.17***.15.17*** CBITS -.03*** -.35*** -.31*** -.34***

30 Leader Perceived Organizational Climate on Practice Attitudes Intercept (grand mean) 18.73*** 2.56*** F b Level 1: Practice (effect coding) MAP 47.85***.43*** CPP 4.42*.16* Triple P SS TF-CBT 54.43***.45*** CBITS 46.37*** -.94*** Level 2: Leader Characteristics ORC Staffing 4.00*.18* ORC Cohesion ORC Stress OCM Autonomy OCM Performance Feedback *p<.05, **p<.01***p<.001

31 Organizational Climate on Leader Attitudes ORC Staffing ORC Cohesion ORC Stress OCM Autonomy OCM Performance Feedback Intercept 3.27*** 3.26*** 3.29*** 3.27*** 3.25*** b Level 1 x 2 TF-CBT.13***.11***.13***.18***.06*** CBITS -.26*** *** -.38*** -.33*** MAP.11***.10***.12***.17***.06*** CPP.05*.04*.04.08* -.01** Triple P SS *p<.05; **p<.01; ***p<.001

32 Summary Therapists and leaders differ in their attitudes towards practices Triple P: expensive; structured protocol MAP: less costly; flexible; significant documentation required Therapist burnout and perceived organizational climate related to more positive attitudes for some practices but more negative attitudes toward others Leader perceptions of organizational climate related to more positive attitudes for some practices but more negative attitudes toward others Implications for considering the views of both providers and leaders in EBP selection, within the context of their views on their workplace

33 Implementation Supports Associated with Leader Attitudes Service System Agencies Therapists Intervention

34 Implementation Supports within Community Agencies Implementation supports (e.g., ongoing training, staffing resources) are critical to the successful uptake of evidencebased practices (EBPs) in community mental health settings. Leaders often involved in selecting and implementing supports but scant research on use of supports in community settings and how support use relates to EBP attitudes. Research Questions To what extent are implementation supports used in community agencies? Do implementation supports differ by practice? What is the relationship between implementation supports and leader attitudes toward EBPs?

35 Sample Characteristics Variable % (N = 163) Reporting Status Top executive leader 1.9 Report directly to top executive leader 14.9 Report to leader directly below top executive leader 36 Multiple levels of leadership between me and top executive leader 47.2 Therapist Reporting Status Therapists report directly to me 61.6% Therapists report to staff that report direct to me 25.8% Multiple levels of leadership between me and therapists 12.6% Therapist Contact Daily 82.6% Weekly 14.9% Monthly 2.5%

36 Methods Agency (N = 58) Leader (N = 163) Practice 1 Practice 2 MIXED Model Level 1: Practice Implementation Supports Attitudes toward EBPs Level 2: Leader Level 3: Agency Practice 3

37 Current Use of Implementation Strategies Very Great Extent 6 Clinical Supervision Supports Organizational Resource Supports 5 Frequency/Extensiveness Not at All 0

38 Multilevel Exploratory Factor Analysis Support Item ICC Within Level Loadings Between Level Loadings In-house staff provides supervision (.05).49 (.06).50 (.16).07 (.25) Outside consultant provides supervision (.08) -.11 (.08).23 (.14) -.32 (.19) Live observation (.02) -.13 (.04).51 (.09) -.02 (.14) Recording review (.04).02 (.04).61 (.1) -.10 (.19) Client progress monitoring (.03).43 (.05).56 (.12).15 (.23) Role play/skills practice (.04).14 (.06).85 (.11) -.15 (.27) Group peer consultation (.03).27 (.05).34 (.11).27 (.16) Use of monitoring tools (.03).05 (.07).70 (.14).07 (.28) Therapists given fidelity feedback (.02) -.00 (.06).73 (.14).11 (.28) Materials (.03).52 (.02).15 (.12).55 (.12) Equipment (.04).37 (.02) -.00 (.06).61 (.08) Facilities (.03).54 (.03).08 (.1).46 (.08) # of trained therapists to sustain delivery (.02).73 (.02).03 (.13).68 (.09) Therapists received initial training (.01).74 (.02).04 (.11).64 (.08) Therapists received initial certification (.02).78 (.02) -.04 (.1).74 (.08) Therapists maintained certification (.04).79 (.02) -.01 (.08).79 (.07) Two within and two between factors: χ 2 = , CFI =.82, RMSEA =.08, SRMR: W =.07, B =.07

39 Implementation Supports Differ by Practice Very Great Extent 6 6 Clinical Suport Scale Item Mean * * * * Organizational Support Scale Item Mean * * * * * Not at All 0 CBITS CPP MAP SS TF-CBT Triple P 0 CBITS CPP MAP SS TF-CBT Triple P *p<.05

40 Model Predicting Leader Attitudes Both scales are significantly associated with leader attitudes toward the practices (controlling for practice). For each additional point on the clinical support scale, attitudes increase by.15 units. For each additional point on the organizational support scale, attitudes increase by.31 units. Item b t CBITS CPP MAP SS * TF-CBT * Triple P Clinical Support Scale.15 (.04) 3.86* Organizational Support Scale.31 (.05) 6.05* *p<.001

41 Summary Use of implementation supports varied by type of support Case discussion and client progress monitoring were utilized more frequently than live observation, recording review, and role play in supervision. Common for agencies to have an in-house supervisor but less common to have an outside consultant provide supervision. Number of therapists who received ongoing training less extensive than other organizational supports. Clinical and organizational supports differ significantly by practice. Both the clinical and the organizational support scales are significantly associated with leader attitudes toward PEI practices Future research will examine direction of relation between implementation supports and attitudes and whether agency characteristics impact use of supports.

42 Concluding Thoughts Practice Characteristics Role in Agency Attitudes Organizational Climate Implementation Support

43 Thank you Funding Source: NIMH R01MH100134

44 Attitudes Differ by Practice F b Intercept (grand mean) *** 3.29*** TF-CBT ***.45*** CPP 29.73***.28*** Triple P 11.77**.16** MAP 5.28*.08* SS 11.47** -.12** CBITS *** -.84*** *p<.05; **p<.01; ***p<.001

45 Therapist Perceptions of Organizational Characteristics OCM Mean of Burnout Items Mean Agreement Burnout 1 Autonomy Performance Feedback Involvement

46 Leader Perceptions of Organizational Characteristics 5 ORC 4 OCM Average Rating Average Rating Staffing Cohesion Stress 1 Autonomy Performance Feedback