The Hidden Power of Kirkpatrick's Four Levels By Jim Kirkpatrick

Transcription

1 The Hidden Power of Kirkpatrick's Four Levels By Jim Kirkpatrick I just completed a thorough (15 minute) internet review of world myths. I was looking for something in particular, but didnt find it. After completing this white paper, however, two new myths will be added to the Google archives. Ah, but prior to that, in order to satisfy your curiosities, I will provide you with 3 potentially life changing myths that I did encounter. Myth #1: Ants speed up the decomposition process of fallen trees. I am sure many of you have had to take a seat on this one. Apparently, ants feed on living matter, and do not eat dead trees. Myth #2: The White Buffalo Calf Woman from the 1800s North American Lakota tribe was able to turn into an actual white buffalo. Ah, lost a few more of you with this one. I am sorry to have shaken a foundational belief, but apparently there were no credible witnesses to this ever happening. Myth #3: Putting an open box of baking soda in your refrigerator helps to eliminate odors. For those of you left standing after #2, I am sure this is a stunner. Reports are that it was nothing more than a good marketing ploy. My father, Don Kirkpatrick, invented the four levels of evaluation over 50 years ago. They were, and still are, reaction, learning, behavior, and results. The good news is that over these many years, many people have become familiar with the four levels. The bad news is that incredibly few have figured out how to effectively implement them. The three new myths that are about to break out and make the scene have to do with the four levels. Here they are. Myth #4: Using a smile sheet for Level 1, pre and post-tests for Level2, and hope for the best is good application of the four levels. For the short response to this untrue myth, this is far and away the typical way the four levels are implemented, even in Fortune 500 companies, and those who rank high on various top training company lists. Myth #5: Kirkpatricks four levels are useful only for course and program evaluation. I have talked with my dad during the past few years about this. While his original work with the four levels was designed to evaluate a supervisory training program, we have discovered that the four model is just as effective with strategic training needs. Myth #6: Kirkpatricks four levels are quite distinct and separate from instructional design, competencies, and performance management. Not true. Much of my recent work with colleagues at Specialist Management Resources (SMR), a global company specializing in providing learning and performance enhancements to organizations, has to do with alignment. These alignments include evaluation with strategy, evaluation with instructional design and strategy, and evaluation with

2 competencies and performance management. These powerful alliances add new depth to the four levels, and all add up to strong synergies, bringing the business partnership model to life, and the effective execution of strategy. The title of this paper is The Hidden Power of Kirkpatricks Four Levels. I chose that title because of the tendency of these myths and other misunderstandings that have greatly limited the power of the four leveis to maximize and demonstrate training effectiveness. I think the best way to proceed with debunking these myths is to present how I see the four levels in relation to todays business and training needs. I strongly suggest that as you read the following pages, you consider and leverage some of these arguments the next time you need to formally or informally build a business case for learning, training, and/or evaluation. Principle 1. There are now six solid reasons to evaluate. Dons original work suggested there were three reasons to evaluate. However, I went back and read his words and he actually said in his 1993 book Evaluating Training Programs (1st Edition), Here are three reasons [to evaluate]. He and I have had discussions recently about new applications of the four levels. When I said I thought there might be more, he got a little wide-eyed and asked me what the fourth one was. As we kept discussing, eventually we got to six. I am not sure he will document all six in the 4th edition (no plans for it yet), but he did nod his head and provided wise input on most of them. Anyway, here are six reasons to evaluate: 1. To determine whether a program should be continued. This should be a no-brainer but, alas, using evaluation in this regard is sporadic at best. The belief that more is best tends to rule the day, and retiring courses seems to happen by default, not design. In order to accomplish this, I suggest you a) develop a matrix to see which of your current courses are aligned with particular organizational directives, b) gather Level 1 data to see if the program material is still relevant to current practices, c) gather Level 3 data to see if people are actually using what they have learned. 2. To improve a program. This is the most commonly used application of four level evaluation. Here, it is relatively easy to match evaluation data and information to program intent and objectives (which are often not in good alignment). This process is best done with a feedback loop of data/information from Level 1 surveys and focus groups, all manner of Level 2 methods, both during and after the program, and from Level 3 surveys and behavioral observations. The idea is to look at the data and determine if there are any snags that break the chain from the learning process to the desired/expected results. Start with Level 1, and work up to Level 3 to uncover, diagnose, and correct disconnects. Doing a good job with these 3 levels will help contribute to debunking Myth #4. 3. To ensure learning compliance. This wasnt around much when Don did his original work in the 1950s, but it surely is today. Some view evaluating learning for compliance purposes as a necessary evil, but we all know that it must be done. Efficiency is the key, and I encourage you to do whatever you can to keep all or most training from being mandatory, as that will discourage individual ownership for learning and keep you from offering more strategic and rewarding courses and programs. As you might expect, this one is all about Level 2 testing.

3 4. To maximize the value of training. This is a bigger, and definitely falls under the heading of the lost art of evaluation. Senior leaders are oftentimes clueless as to how learning contributes to bottom line results, and learning professionals arent much better off in being able to convince them. But before we get to demonstrating value, we want to make sure first that we are adding value. Here is where Level 3 comes in front and center. As most of us know, a never-ending challenge is to get learning to stick. I wrote a book entitled, Transferring Learning to Behavior (Berrett- Koehler, 2005) where I describe the importance of and means of going about ensuring that Level 2 learning is applied as Level 3 on-the-job behaviors. Mission critical or non-negotiable behaviors must be reinforced through Level 3 evaluation surveys, observation of behavior and work, focus groups, interviews, action plan monitoring, action learning or it just wont stick. And if it doesnt stick, all of your precious training will be for naught. A very important principle of level 3 is to be sure to use it to determine if application gaps are due to the training itself, or the environment the participants come back to. By the way, this principle applies not only to courses and programs, but also to strategic evaluation. This is done by the use of aggregate Level 3 data used in dashboards and scorecards. And as an added bonus, your successful execution of this purpose of evaluation will help us all blast Myth #5 out of the water. 5. To align training with strategy. This purpose is also closely aligned with strategic evaluation. The basic premise is to ensure that training is aligned with the expectations for particular programs and curriculums. The main reason this is so important is because of the increased pace of change in business needs and expectations. This not only connects with organizational needs, but with individual and occupation needs that need to feed business needs. This purpose can only be accomplished if an effective needs assessment has occurred with key stakeholders to uncover specific expectations, so that learning and new behaviors can lead to the accomplishment of those expectations, and so you can use them as primary Level 4 metrics. To accomplish this purpose, I suggest you view Level 3 and 4 data and information in light of behavioral objectives and desired outcomes as delineated by stakeholders. This purpose also helped lead me to use the word hidden in the title, as it is seldom dealt with effectively. 6. To demonstrate the value of training. This is the one that Don talks about in terms of justifying our existence as training professionals. While improving programs involves a feedback loop, this one is linear. By knowing the audience that you are trying to demonstrate value to, you can gather data and information accordingly (at each of the four levels sequentially) and present to them strong evidence that indeed effective training has led to targeted learning which has led to critical on-the-job behaviors which has made a positive contribution to the bottom line (i.e., stakeholder expectations). While this can be done effectively with programs or even individual courses, it is even more effective from a strategic point of view. And if anyone asks you, Is it ok if we skip Level 3 when we are demonstrating value?, you answer then, Of course not. Everyone knows that one plus two does not equal four. Principle 2. Evaluation should be deliberate and purposeful. Once you decide on what purposes you want to evaluate for, you will need to decide on the techniques you will use (the methods to collect the data and information), and the tools that

4 you will need to do it. Once you make those decisions, it is time to deploy resources to develop the tools, administer them, analyze the information, and make recommendations. With all the work that is involved with this, and as busy as people are, you can see why I strongly recommend that evaluation be deliberate. This will allow for the best use of your limited resources, and will not cause people to be over surveyed. A competency model fits nicely with this principle. These competencies should be mapped to specific course/program content, and to your performance management model to ensure a balance of efficiency and effectiveness. Principle 3. The business partnership model is a cornerstone of Kirkpatrick Four Level evaluation. There is way too much us and them in the world of training and business. I believe there is equal fault in that. First, business leaders too often see training as separate and distinct from anything that brings bottom line value to the organization. That is rather ignorant thinking, but is what we have to contend with. Second, the rest of the fault is ours. Instructional designers spend most of their time with the DDI components in the ADDIE Model (Analysis, Design, Development, Implementation, Evaluation). They are comfortable with course development and delivery, and pay much too little attention to the needs analysis and evaluation. So, guess what the business leaders see? Courses and programs being conducted. Guess what they dont see? Good partnering in relation to needs assessments (A) and strategic evaluation (E). Also, the ROI craze is beyond me. I think few major developments in the field of learning have done as much to re-emphasize the us versus them mentality as that. Kirkpatrick four level evaluation is all about the business partnership model between business and training / learning. While I will not go into the details about how to develop it, suffice it to say that there are numerous ways to build that partnership prior to training, during training, and after training. The close integration of program design, competencies, and performance management all fit under this concept. Principle 4. Level 3 is the forgotten level, yet is the key to maximizing training effectiveness. Since I call Level 3 the forgotten level, even though I mentioned it above, I thought I had better mention it again in case you forgot. I also call it no mans land (man being male and female) because since training people are so busy with their precious training programs, and business leaders are all-consumed by getting results, the reinforcement of critical onthe-job behaviors gets left to the individual. Any of you who know me know that Level 3 is my passion. After all, what good is training and learning unless it is applied? Principle 5. Return on Expectations (ROE) is the key to setting up training success through the determination of Level 4 metrics. Now we are really getting after debunking Myth #5. I believe that effective training begins with the end in mind, targeted learning occurs by beginning with the end in mind, and learning makes significant contributions to the bottom line by beginning with the end in mind. Using the business partnership model, and the A in ADDIE, uncovering these expectations sets the table for a) effective program development, b) the ability to tie training to results, and c) happy (with training) business leaders. Once these expectations are clearly defined, they can then be converted to Level four metrics that we can then work

5 towards. An added benefit of attending to this need is that it is relatively easy to identify situations where additional training is not the answer to the business need. There is an additional concept that is worth noting here. They best way to do this is through the process of managing expectations. It is not enough to uncover and clarify needs from stakeholders, than report back to them 6-12 months later. One must keep in close contact with them along the way, sending progress reports that include leading indicators, then check to ensure you are on target. Principle 6. Gathering comprehensive data and information to create a chain of evidence is the best way to demonstrate training value to stakeholders. I have a good friend who is an attorney. In fact, part of his job is to work with the whole science of knowing the jury so attorneys can better discover, develop, and present evidence that will compel them to find in favor of their case. Guess what? It is exactly the same thing with learning professionals and business leaders. Don, my friend, told me that most jurys are moved by a combination of expert testimony and weeping widow testimony (his term). Again, as training professionals, we need to read our jury, know what kind of data (akin to expert testimony) and information (akin to weeping widow testimony) we will need to convince them of our case, go get it, prepare it, and effectively (and at times passionately) present it. Principle 7. Now you need to act on what you have been told. I have presented to you my thoughts about what is current and cutting edge in the world of Kirkpatrick Four Level evaluation. I trust that something in preceding pages has struck a chord of something that would work in your organization. All I ask is that you follow up with that chord.