Measurement of implementation components in empirically supported treatment programs Nordic Conference on Implementation of Evidence-Based Practice

Transcription

1 Measurement of implementation components in empirically supported treatment programs Nordic Conference on Implementation of Evidence-Based Practice Linköping, 5-6. February 2013 Terje Ogden The Norwegian Center for Child Behavioral Development Unirand, University of Oslo 2/13/2013 Slide 1 Background of the study Ten years after the nationwide dissemination of two evidence-based treatment programs (MST & PMTO), the status of the implementation components was evaluated in a cross-sectional study. The aim of the study was to pilot a standardized measure of implementation components by examining the factor structure, the reliabilities of the scores, and their association with implementation outcome variables. The aim was also to compare implementation profiles of the two evidence-based programs based on multi informant assessments. 2/13/2013 Side 2 1

2 2/13/2013 Side 3 2/13/2013 Side 4 2

3 Core implementation components Supervision/ coaching Performance assessment Integrated and compensatory Decision support data systems Facilitative administration Training Practitioner selection Systems intervention Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 Leadership 3

4 Implementation drivers Recruitment - practitioner selection - how staff is recruited to work with the program during its initial phases of implementation, and how staff is recruited to help sustain the program over time. Training - activities related to providing information, instruction, or initial skill development to practitioners and other key staff in the implementing organization. Supervision/coaching - how supervisors provide guidance to therapists and practitioners and also how often, where, and how feedback information is collected. Performance assessment - the type, frequency, and method of performance evaluation of the practitioners in relation to their use of the program. Specifically, questions deal with measures of integrity to the method, as well as how often and by whom their performance is evaluated. 2/13/2013 Side 7 Implementation drivers II Decision support data systems - the availability of information through systematic acquisition of data in order to provide feedback to stakeholders, therapists, coaches, and policy makers inside and outside of the organization. Facilitative administration - whether those in charge of implementation in the host organization had restructured and adapted the organization to make implementation and sustainability successful. Systems interventions the participants organizations work to influence the systems and policies in their region to develop better support for the innovation. Leadership - the ways different people within the organization engaged in leadership behavior and provided systematic support, clear communication with practitioners, provided decision-making, garnered feedback, and engaged actively in a manner that was conducive to successful program implementation. 2/13/2013 Side 8 4

5 Method and participants Structured interviews adapted from the Measures of Implementation Components of the National Implementation Research Network Frameworks (Fixsen, Panzano, Naoom, & Blasé, 2008), All MST therapists (N=56), their teamleaders (clinical supervisors) (N=21) and managers (N=11) were interviewed, A random selection of 93 PMTO therapists, their supervisors (17) and organizational managers (20) were interviewed. 2/13/2013 Side 9 Results MST and PMTO had different implementation profiles and therapists, supervisors, and managers evaluated some of the implementation drivers significantly differently. The factor analysis of the eight one-dimensional subscales resulted in an individual clinical-level factor and an organizational system-level factor. Age, experience, and number of colleagues in the workplace were negatively correlated with positive ratings of the implementation process, but the number of colleagues working with the same program predicted positive ratings. 2/13/2013 Side 10 5

6 Mean scores for the informant groups 2/13/2013 Side 11 MST and PMTO therapists compared (means scores) 2/13/2013 Side 12 6

7 Factor loadings for implementation component scales SCALE INDIVIDUAL CLINICAL LEVEL Supervision/coaching.86 Recruitment/selection.78 Performance assessment.74 Facilitative administration.61 Leadership.58 Systems intervention.50 Decision support data system Factor loadings <.40 not included ORGANIZATION AND SYSTEM LEVEL 2/13/2013 Side 13 Competency and organizational components Performance Assessment (Fidelity) COMPETENCY ORGANIZATION Selection Training Coaching Integrated & Compensatory Leadership Systems Intervention Facilitative administration Decision support data system 7

8 Implementation quality 10 years after Overall, the respondents reported mean scores at the upper end of the competency scales, Both programs seemed to have established and sustained highly structured and consistent procedures for recruitment, training and supervision. High ratings were also given for working with external systems and leadership. Lower mean ratings were given for the organization scales which were under the control of the local agencies and represented new demands made by the implementation framework. (performance assessment, decision support data systems and facilitative administration). 2/13/2013 Side 15 Integrating policy, research and practice Information, dissemination and training, no matter how well done, is by itself an ineffective implementation method. Implementation components at the policy and organizational level is vital for the sustainability of large scale implementation efforts. Examining indicators of long term sustainability of evidence based programs highlights the importance of: Policy: Maintain funding and support, Practice: Establish and maintain quality assurance procedures, handle turnover of staff, and train new interventionists, Research: Build, maintain and update the knowledge base of effective empirically supported interventions and their implementation. 8

9 Open access: 9