Scaling an Effective Intervention: How to Build Implementation Capacity, Replicate with Fidelity, and Produce Consistent Outcomes

Transcription

1 Scaling an Effective Intervention: How to Build Implementation Capacity, Replicate with Fidelity, and Produce Consistent Outcomes Ingvild: Topic how to scale effective intervention. This webinar is co-sponsored by the partners of performwell.org which are research organizations: Urban Institute and Child Trends, as well as performance management software company, Social Solutions. My name is Ingvild Bjornvoid and I m the Director of Strategic Initiatives with Social Solutions and today I have the pleasure of being joined by Dr. Dean Fixsen. I know that many of you are going to want to access the slides and the recording of this webinar. Both are going to be made available to you. Everyone who registered will receive a link to the slides and recording within the next couple of weeks. If you want to access them earlier you can go to performwell.org and click on webinars and the slides and recording should be available there by the end of this week. We are going to take questions towards the end of the webinar today but you are welcome to submit them at any time. Questions can be typed in to the chat feature in the lower left corner of the screen. We have reserved quite a bit of time for questions and will take as many of them as we can. Moving on to our topic today, when is constant implementation science? Some of what is the most striking I think is how much is actually known about what it takes to scale programs effectively. Yet, how little of what is known is actually put in to practice typically. For such a long time the conversation has been very focused on figuring out what works so much so that how to

2 actually implement evidence-based programs and practices in real life is something that s gotten pretty little attention. Of course knowing what works helps us very little if we are unable to replicate it. If this webinar is any indication however I think [02:00] the tide is turning because this event is turning out to be one of the most popular events in Perform Well s two and half years history, and I image that is a reflection of an increasing interest in this topic in general which is fantastic. It is my great pleasure to welcome Dr. Dean Fixsen who has done some very exciting work in the field of implementation. He has spent his career developing and implementing evidencebased programs, initiating and managing change processes in private organizations of service delivery systems and just in general working with others to improve the lives of children, and families, and adults. Dean is coauthor of the highly regarded monograph Implementation Research: A Synthesis of the Literature. He served on a number of editorial boards and advised federal, state, and local governments, and he is a senior scientist at the Frank Porter Graham Child Development Institute at the University of North Carolina, Chapel Hill. He is also co-founder of the National Implementation Research Network as well as co-director of the State Implementation and Scaling of Evidence-Based Practice Center and president of the Global Implementation Initiative. Dean, it is an honor to have you with us today. Welcome. Dean: Thank you very much. I appreciate it. Thank you for the very nice introduction. You can see from the picture that was up there that I look like a very old gray hair person but I m really only 35 years old. It s all that work that s actually getting me to look so old. Today we want to talk about implementation in practice. This is a subject that s very near and dear to our hearts because you all as practitioners, the people working in agencies and organizations [04:00] doing the work that is designed to produce good results for children, and families, and adults, and communities every day, this is really where implementation science and practice and policy is now developing in such a way that it can be very helpful to you. What I d like to do in the next 30, 40 minutes here is just to give you an outline of some of the important features of implementation from a practice point of view. We ll get into some of the details of this, some things that you can go back and try this at home tomorrow because they have practical implications. As you re listening to this feel free to put your comments, questions, concerns, challenging, You said this and I don t believe it at all, any of that kind of thing, put it in the chat box and we will get to as much of that as possible. We ve tried to reserve as much time for your questions as possible. This first part is really the overview. As you can see here implementation in practice, myself, Karen Blase have been the codirectors of the National Implementation Research Network. We have moved out of those roles. Alison Metz and Melissa Van Dyke now are the co-directors of the NIRN so you will hear more

3 about them from them as time goes on. The big thing with implementation science and practice boils down to the formula for success. This simple formula is the result of about 50 years [06:00] of very hard work, but we re coming to the point where we understand what the big sets of activities are that contribute to those socially significant outcomes that we all want to see. Effective innovations times effective implementation times enabling context, very important. Effective innovations, as you know there s been a big emphasis for the last 25 years on defining evidence-based programs, and practices, and approaches. All of those would fit into the effective innovation category. There is some nuances to that that we will discuss here in a minute. Effective implementation is something that we re learning about. Enabling context, for the first time in my many years in the field we now have government, state, federal, local governments interested in knowing what they can do to support us as we are working in agencies to accomplish these socially significant outcomes. What happens though, notice that these are multiplications. It s affecting implementation times innovation times enabling context. We know that in any multiplication problem if any factor is zero the outcome is zero. So why is that we have these trillions of dollars really that have been invested in developing evidence-based approaches and yet we still have problems with literacy and education? We still have problems with delinquency. We are not achieving those socially significant outcomes. We think that it s become that center box there, the effective implementation box, has [08:00] not been filled in the way that it should be. This is where you and we come in together to define what effective implementation is so that we can achieve socially significant outcomes more often. We have studied this for many, many years. Karen Blase and I have worked in as director of agencies. We worked in child welfare and mental health, judicial justice assisting children and families. We also have been studying and doing research and evaluation work with respect to this, so we have a foot in each camp where we are practitioners of evidence-based programs and of implementation, and we re also scientists and researchers who are studying this. Out all of that we have arrived at we call the active implementation framework. Just a quick review. Any kind of implementation project starts with what it is you re trying to do? What are you trying to implement? Is this a literacy program? Is this is a community prevention program? Is this is a delinquency reduction program? So what is that? You notice that is says usable intervention. Usable intervention is different than an effective or an evidence-based intervention. Evidence-based interventions as you know there s been a great debate and a pretty defined definition now of what constitutes rigorous evidence and support of an intervention. But unfortunately all of that rigorous research has not actually defined the intervention itself [10:00] in such a way that makes it teachable and learnable and doable in practice. A usable intervention, you can go on our website and find the criteria for that. A usable intervention is slightly different, still effective, you still want it to be worthwhile, but there is a different set of criterias. How then are you going to go about getting that intervention into practice? These are the implementation drivers. We re going to spend a fair bit of time on those today. We also then have improvement cycles, another way of whatever intervention, whatever innovation we have, however well we do the implementation drivers, we know that we only know what we know today, we re only as good as we are today, and those are probably better than it has been in the

4 past but still not what it should be to maximally effective. Improvement cycles are very much a part of implementation science. How to go about that in a deliberate and purposeful way is very much a part of the active implementation framework. Another category of activities is who then does this work? Who knows how to create a usable intervention when the critical components of that intervention have not been specified? Who knows how to make good use of the implementation drivers? Who knows how to make use of the improvement cycles so you can rapidly like within the next 20 days solve problems and come up with better solutions to things? The answer to those questions is implementation teams. They are the ones who do the work of implementation. [12:00] Finally we have when are these things done? There are stages of implementation and there are times where you are exploring things, there s times where you re getting the resources together but you still haven t done the innovation itself. You re finally ready to start doing the innovation in a new way, new way of work in your organization. You know how anything is, whether you re learning to ride a bicycle for the first time or God forbid learning to play golf. I mean it s not pretty. But you have to get through that initial implementation part to get to the point where it s skilful and you re confident in how you go about it. Stages of implementation are very important. We are not going to go into all of these. These are on the website. If you google N-I-R-N, NIRN or you google A-I hub for Active Implementation Hub you will find lots of information about all these things. Here are the implementation drivers, and this is where we re going to spend our time today because this is the closest to home. This is the stuff that you interact with probably everyday in your organizations and in your agency. What we want are those reliable benefits to the children, families, adults, community, neighborhood members that we re dealing with. We want to have a way to reliably help improve the lives of other people or to prevent problems from occurring in the first place. To do that it turns out that we need to consistently use these effective usable innovations. [14:00] But how does that happen? That s where the implementation drivers come along. Innovations by definition are new ways of work. If it s a new way of work then we, the practitioners and managers, we need to learn how to do that new way of work. The competency drivers on the left hand side are critical. Selection, training, coaching, performance assessment, those are very important to have build into agency routine so that we can routinely produce people who are practitioners operating at a high level so they can consistently produce those reliable benefits. We found over the years that that is necessary but not sufficient. People could learn this, become quite confident to achieve all the fidelity standards, produce wonderful outcomes, and that would last for a while. What was missing were the other 2 parts of this, especially on the right hand side where it says organizational drivers. It s not only practitioners who have to change their practices to use the innovation. Organizations have to change their policies, procedures, their ways of work in order to support practitioners doing the innovation as required and to support the implementation components. Who s going to do selection and training and coaching in new ways to support the development of competency among practitioners? Then the leadership drivers. You can t read anything about implementation that doesn t say leadership is very important. But it s only in the last 10 years that leadership itself is now [16:00] operationalized, defined well enough that we know how to

5 detect leadership skills in the environment. We know that there are different kinds of leadership called for at different times. Again, these are the drivers. Now this is a critical thing because here in this slide you can see that the fidelity part is missing. I will say that as we have worked across human services in the United States and globally fidelity is rare, fidelity assessments are rare. Most evidence-based program developers have not created a fidelity assessment. Most organizations do not have fidelity assessments. But notice without that we have now disconnected all of the competency and organization and leadership drivers from the consistent use of innovation by practitioners. We no longer have a clear link between implementation activities and innovation uses and innovation outcomes. We re going to spend some time on fidelity here just because it is such a critical thing and it is an essential component and it s pretty much a missing component. This slide shows you what things look like. We ve shown you the clean version but when these implementation drivers are being used in an organization you have people who were doing multiple roles, you have multiple skill sets represented in folks. This is all a way of [18:00] integrating. As you noticed in the middle it says integrated and compensatory. This is the way of keeping it integrated. When it comes times for training you have a trainer who is organizing the training session, making sure that the content is there and the room is setup and people are prepared to present their sections, but the people who are providing the training, coaches provide training, administrators provide training, leaders in the organizations provide trainings, lots of people provide training. Training is not done just by one person. It s done by a bunch of people. Who has a big stake in training is coaches, because whatever is not taught during training, guess who gets that practitioner next to develop their skills. Well, it s the coach. This integration data selection come from training, come from coaching, go into the decision support data system on the administrative side so that that organization driver then, the first users of that information are the people who are making decisions about selection training and coaching, about how well the organization is supporting practitioners. You can see how these things are highly integrated. People are performing multiple functions, so don t get the idea that all of this is separate activity. I dare say in any organization of any size all these activities are going on already in one way or the other. Somehow the other people are getting hired so the selection driver is there. There s some kind of orientation, if not training. You have supervision if not coaching. You always have the managerial [20:00] supervision side of things and directors of it. There s always a data system. You re tracking money if nothing else in the data system. All of these things are not unusual activities for the organization. The unusual part is that it s focused and integrated to be focused on the use of innovation with fidelity with those consistent uses and with those reliable outcomes for people. It s how we go about these things and we re going to dwell on that for just a bit. Here I wanted to just talk briefly Here is the place where we began. We have effective innovation practices. We have information of one kind or another. It doesn t have to be a randomized control trial. You may have data right there in your own organization that says when

6 we do things this way we produce the outcomes that we want. Those effective innovation practices are tied to those outcomes of the innovation. Well, who is it that produces those, makes use of those innovations? Well, practitioners, staff, administrators, boards as we were just describing, they are the people who are responsible for assuring that those innovation practices are effective and are producing the desired outcomes. There is a picture of the But now that begs the question, Well, who prepares all these practitioners and staff to do this? This is where the next part comes in. This is where implementation processes, implementation practices made use of [22:00] my implementation teams. Implementation teams don t interact with the children and families. They interact with the practitioner staff and the organization that is there. The use of the drivers, the use of exploration, installation, those of the implementation stages, those are the critical features that are there for implementation. Now just as we have fidelity measures for innovation, there are also fidelity measures for implementation. Sometimes people will talk about implementation fidelity when they really mean innovation fidelity. For us implementation fidelity means that we are using the implementation drivers and the other active implementation frameworks as intended, and there are assessments of that that we make use of. But the critical part of this now, we now know what it is we want to do, we know how we want to do it with the implementation part, we know who is going to be helping to facilitate all of this, and that answer is the implementation team. This is what we call a cascading logic model. You can see the cascade right here. Each thing is linked very directly to the next. Who makes use of innovations? Well practitioners, staff, administrators. How do they learn how to do that? Well, that s the work of implementation teams and so on. But to know that these things are linked in that way and that you have assessment and performance of each is a critical feature of implementation. [24:00] Does it matter here if we have an effective innovation evidence-based or designed in some way to show that it s effective? We have the support of an expert implementation team. You can get about 80% success, in this case meaning 80% of the organizations that are being worked with by that expert implementation team are able to meet high fidelity standards within 3 years, maybe 4 years. But without that there s a growing demand of data now that you get about 14% success in 17 years. Does all this implementation stuff make a big difference? Well, yes. In the availability of these expert implementation teams in organizations, in human service systems is critical. We are spending a huge part of our time right now in the implementation of scaling up of evidencebased practices in education and it s really around these development of these implementation teams is what we re doing in large scale education systems where you have a million students being taught by 100,000 teachers. Very, very important. Another bit of data on the implementation team, Alison Metz has taken the measures and we re about ready to review some of those measures here in a minute, and has assessed those in practice. This is in a county here in North Carolina. It s in a child welfare environment. They ve spent the first year or 2 working on developing what they called a success coach that s the [26:00] innovation and demonstrating the effectiveness of that. During that time they were also beginning the development of the implementation team.

7 The team 1 time 1, here you see on a scale of zero to 2, zero not in place, 1 partially, 2 fully in place. That implementation team by about year 2 and half or so is when this first assessment was done. Their average score was 1.1 on this 2 point scale and had about 18% of the case workers making use of the success coach intervention innovation as intended. You see time at time 2 and time 3 the composite scores for the implementation team went up 1.68, 1.83, and fidelity also increased from 18% to 83%. This is really the first indication that we have. These are the first data that exist, that demonstrate a pretty direct link between what an implementation team does and the outcomes that that team is achieving with respect to fidelity. Fidelity as we noted earlier is a critical feature. Let s take a look at this for a second. Here s Functional Family Therapy born in the 1970s, been around for a long time, well used in the fields of delinquency and some mental health applications. But down the left hand side the numbers 1 through 25, those are individual therapists who have been trained and receive some support [28:00] for their work as functional family therapists, so they re working with families. You can see that 12 of the therapists were functioning above the fidelity criteria. They worked with 204 families, 13% recidivism. The remaining 13 of the therapists were deemed to be below the fidelity criteria that was desirable. They worked with 200 and some families as well with 28% recidivism. You will notice here in the middle is the control group with 22% recidivism. The lack of fidelity here you notice is not a neutral event. It actually produced worse outcomes than the control group. Those kids and families were done no favors by the people. But everybody was using functional family therapy, so it s the same program supported in the same way and different results. Now this is the information then that goes into the decision support data system. Who are the people who will look at this right away? It s the implementation team. They will start looking at this and say, Well, why is this? Why do we have only 12 out 25 who are meeting these fidelity criteria? What s going on here? They will start looking at various possibilities. For example, it could be that when the implementation team is looking at this they say, Well, 1 through 12, they were all coached by Karen Blasé. These ones [30:00] numbers 13 to 25, Dean Fixsen was the one who was their coach. We don t have a functional family therapy innovation problem. We have a coaching problem. Dean needs to learn how to be a better coach. It directs your attention to where a solution may reside. It could be that you analyze this and 1 through 12 were trained by 1 group of trainers, and the others were trained by a different group of trainers. Well now we have a training program and so on and so on. You can imagine the hypothesis that gets generated. But without this data what would you conclude? You would conclude that functional family therapy does not work because overall the recidivism rate for all 25 of these therapists is no different than the recidivism rate for the children and families who were in the control group. Very, very important to have fidelity and to know this on a practitioner by practitioner basis. This is where the problem solving occurs. Very critical and yet we don t have much of this data in any organization. Now we want to move into some of the details of this. If you go on to the Active Implementation Hub I ve put the url there for the assessment that we are about ready to go through. But here in

8 the resource library which is the tab that s clicked for this one there is a lot of information. The Active Implementation Hub, we ve been working on this for the last few years and finally in this last year it s come together very nicely. [32:00] Lots of information about active implementation, what it is, how you do it, their exercises, pretty cool stuff. I really urge you to go there. But this assessment that we re about to go through is here at this particular url that s highlighted. Let s start looking at the details. This is the stuff that you can start going back to your home agency, and start thinking about this, and start working on these things tomorrow. There is implementation when you get right down to it. It is a lot of hard work. You have to create some new things in the middle of all the things that are going on in the organization right now. As we say in education we re trying to create a new system, but at the same times those school busses pull up every day and all those kids get out, they run into those classrooms, so the new has to be build in the context of continuing to do what we have been doing. At some point that switch is over but to get started takes quite a bit of determination and effort on your part. We re just going to look at the competency driver, so left hand side of that implementation driver s triangle. But here are the items that we look for. Here is the capacity that we want to build in every organization that we re working with. Every one of these starts out with somebody who is accountable for this particular driver. Somebody has to be responsible for organizing the scheduling for [34:00] conducting fidelity assessment. We re talking about performance assessment if you notice at the top. The scheduling of these, the preparation of people and the improvement of the fidelity assessment over time, that accountability has to be lodged with a person. Each of these starts out with accountability that s clear. There is a lead person who s designated to assure that these things happen and that they improve over time. But with any of these, any implementation activity needs to be transparent. There are no secrets here. We don t want people to be surprised by things. Even though we know they probably will by the time we get to that part of things, that they will have forgotten that they agreed to it early on. But still, we want everything to be transparent. For performance assessment and for the fidelity assessment there needs to be some early orientation of practitioners and other staff to here are the criteria, here is how we go about doing the observations, here is how we summarize the data, here s how we report right back to you, here s is how it s used by the organization to help all of us to improve. This is not being used in a punitive way. The third thing, performance assessment measures are highly correlated predictive of indented outcomes. This is a difficult criterion to meet. You will not be able to generate this kind of information right away. But as the decision support data system, as the data accumulates there you will have more information about fidelity, you will then have more information [36:00] about the proximal outcomes, and you will be able to generate things like we just reviewed for the functional family therapy therapists. You will know they re fidelity, you will know their outcomes, and you will be able to compute that correlation. Number 4, assessments are conducted on a regular basis. Setting the schedule is so important. The first one occurs after 3 months. The next one after 6 months, and then annually at the end of the first year, and annually thereafter is not an atypical set of assessment. Other programs have the fidelity assessment every month because the risks involved in having low fidelity

9 performance are so great you cannot afford to have too much time go by without knowing how well the practitioners are doing. But whatever that schedule is it s known to everybody and it s adherent to. It s not like we ll do the assessment when we think the person is ready or we ll do an assessment when the manager says, I think you know so and so is far enough along. No, it s 3 months, give or take a couple of weeks. That s when it s done and it needs to be on that regular basis otherwise it s too easy to start escaping this and you re losing data. You have an efficient and a practical assessment system, so doing fidelity assessments takes a long time and energy and effort. Somebody has to be doing these things. If you have direct observation and direct observation may take [38:00] 10 minutes or it may take an hour, but if you have 50 people in your organization you start adding all that up, and that s a lot of time. So how are you going to design this in such a way that it is practical and doable in your organization? Making sure that you are assessing competence of practitioner performance as well as whether, yes, they have 5 sessions. Well, good, you want to make sure that if 5 sessions are called for that they did all 5. But on the other hand what happened during that session? It turns out that there are now more and more data to saying not only there are the things that we are asking practitioners to do that define the innovation, but there are others things that we re asking practitioners not to do. You can do 5 things really, really well and do this other thing that takes away all the advantage that you just created with those 5. Motivational interviewing and a few others have some data now on them. Use of multiple data sources so you re not just relying on one thing for part of the assessment. Recognition for participation. Here s something that it s interesting, a hard lesson learned over many years, but what you celebrate in staff meetings etcetera is the fact the fidelity assessment was scheduled, the person showed up and was held and so participation in the process is what is talked about. There is no talk about past fail, somebody did better than somebody else. No, no, no. This is all about [40:00] participation in the process because you don t want this to become punitive. You want this to become information gathered to solve problems. So the recognition for participation is key. Then the data are used, and especially used to look at the effectiveness of coaching, which is the first example I gave for F15, for the functional therapy family where Karen Blase was a great coach and produced wonderful outcomes and I was a very poor one and the folks that I was coaching didn t do so well. We find that is not uncommon. Just looking at the time here, we don t have time to go through each and every one of these but you can go to the website and look at these. Here is for coaching for example, the big thing here is number 2. The coaches are fluent in the innovation. The best coaches are the people who meet all of the certification criteria or the fidelity criteria 2 or 3 times. They know that innovation inside and out. They ve used it themselves. They really understand the nuances, so they make terrific coaches. That s probably the biggest piece of that one. When we look at training, training is very interesting when we start to do the assessments, and again building capacity to do all the things that are listen here in each of these. But the big issues here are really around number 3, that training is scaled based. What do you want

10 practitioners to be able to do [42:00] on the first day that they are interacting with a child, a family, a community member, a student in the classroom? It s that set of skills that you want to have at a pretty high level when they leave training. Training needs to have behavior rehearsals or role-plays built into it. We call it behavior rehearsal because people would rehearse the very behavior that we want them to engage in when they walk in the clinical setting. But that means then that you have to have people who are qualified and expert at running those sessions. It s not about stumping the trainee. It s about having the trainee learn the skills. That number 3 is the tricky part of the training driver. Recruitment and selection, we get a lot of discussion about this, how it s outside of the control of people at the clinical level. But we find lots of ways in organizations where the HR departments can still look after the rules and regulations and reporting things. The essential thing here is the interview. Once you get to the interview you re actually interacting with the candidate. This is the critical part. Here it s number 4, the interactive interview process. This is where it requires skill on the part of the interviewer. Again, who wants to be in this interview session to look at new candidates who are about to be hired as practitioners? I ll tell you. Your number 1 person who s going to volunteer to be an interviewer is your coaching staff [44:00] because again who gets hired they re going to be trained, but the coaches are the ones who interact with them on a day-to-day basis to help assure that they re meeting the standards and will meet the fidelity standards that are there. Having these behavioral vignettes, these little behavioral rehearsals, brief, very brief but you re looking to see how quickly does a person start responding to a situation that you ve just presented to them in this little role play. What kind of judgment are they exercising? Do they just freeze up and not able to respond at all, which would not be a good thing if you re interview for home based services where they re going to be out spending a lot of time in difficult situations all by themselves. You really get a chance to see them as they re likely going to be in practice. The ability to accept feedback is underlined there. No matter how well the candidate does we always find one thing that we can offer some feedback and say, do it, great that when you engaged me, I was playing the role of the dad here, and you engaged me so well. But one thing that we really focus on here is giving rationales and reasons for a change. I wonder if we can just do this again and say to the candidate and this time provide a rationale, a reason why it should change the way I m interacting with my son, Okay, let s do it. So are they able to accept the feedback in the first place, and then respond to it in the second place? [46:00] Because what is this like? This is what happens in coaching day after day after day. If the candidate says, Well, that s not what I said. That s not how it went, is that really who you want to deal with for the next 5 years that they re around? Probably not. That s a critical part of the recruitment and selection part of things. Running short of time here, but just to have you see here in education we are now developing assessments. Here is the district capacity assessment, the school district, has many schools which means lots of teachers and thousands of kids who are impacted by what happens. The dark green is the score for this district in 10 of 2013 and the light green is a few months later in June of 2014, so there s about 8 months between these. The green is the baseline assessment.

11 You see a mixture of scores, the dark green. The light green is what happened after we got an implementation team up and running even in a cursory way in that particular district. Here is another district. In this particular district they had a bunch of schools that were in the turnaround status. If you re not familiar with education these are the lowest 5% of the schools in terms of academic performance. But you see on the pre-scores here, these are quite low, 20%, 30%, some are zero performance assessment. If you notice the first thing that we did in the 2 months that we were [48:00] working in that district, the first few months we were working on the district performance assessment went from zero to 100%. It s because fidelity is so important. That is where we started, because that s what would drive change in all the other categories. Bottom line here, pay attention to the active implementation frameworks in your work. Do you have usable interventions? Are you paying attention to the stages? Are you trying to do too much too soon? How about the implementation drivers that we just looked up? Do you have improvement cycles in mind when you are facing problems or needing to create something new? Do you have something like an implementation team, a small group of people of 3, 4, 5 people who meet and go over the information and really think about how to do this work differently? If you liked any of this and you want to go Dublin, Ireland we have the Global Implementation Initiative that many of us are a part of. The next Global Implementation Conference is in Dublin May 26, 27, and 28. The last 2 have been fantastic. The first one was 2011, the next 2013, so it s every other year. But 700 or 800 people and by and large two thirds of the folks who are at these meetings are people just like you, they are people who are crying very very hard to change organizations and systems. We have our fair share of scientists and researchers but it is very much a practice oriented group. With that I m going to stop and turn it back over and I m only 8 minutes [50:00] past where I should be. Thank you.