Quantifying the Organizational Benefits of Training August 2010 The Majority of CEOs Want Return on Investment Data for Training Programs, But Few Receive It Watch List A recent survey of CEOs found that 96 percent of CEOs wanted to see learning and development connected to business impact data, but only 8 percent receive this information. Most organizations spend one percent or less of the learning development budget on measurement. Best practice organizations typically spend between three and five percent of the training budget on measurement and evaluation. According to the 2006 Blanchard Survey, 58 percent of training and development professionals say their biggest training implementation challenge is proving the impact and ROI of training. The return on investment (ROI) of every training event does not need to be analyzed. Events like new employee training or retirement planning do not need to be measured. While ROI is a useful tool, it has shortcomings. For example, it does not do a good job at identifying how engaged learners were during a training course, nor is it effective at measuring how much information employees retain after training. In addition to ROI, there are alternative measures that can be used to quantify training benefits. These include impact on business (IOB) and return on expectations (ROE). Related Reports Justifying Training & Learning Costs Return on Investment vs. Impact on Business Key Takeaways There has been growing interest in applying ROI to the training function in organizations due to increased need for accountability, greater interest in continuous process, and an increased focus on improving training efficiency. The ROI Institute conducted a study which showed that only one in ten chief learning officers (CLOs) measure the impact of training on the business. Yet, nearly all CLOs measure success by how many employees are trained. Since there is so much diversity in the types of training programs that organizations offer, there is no one ROI formula that can be applied to corporate training. Courses that are strategic to the business plan, programs where management has high expectations that training will positively impact the company s bottom line, and training that is offered to large numbers of employees are training programs that are well suited for ROI analyses. Effective ROI analysis processes are simple, credible, theoretically sound, accountable for factors other than training which may affect performance, appropriate for use with many different HR programs, and usable with all types of data. With e-learning programs, hard cost savings, like reducing travel to training sites, are usually easy to identify whereas soft cost savings, like improved information retention, are more difficult to quantify.
Executive Summary Training evaluation and measurement are not new concepts. In 1959, Donald Kirkpatrick developed the Four Levels of Evaluation. With each level of evaluation, trainers gain a more precise measure of program effectiveness. However, each level also necessitates more time, focus, and analysis. The first three levels consist mostly of soft or qualitative measures. In recent years, levels one through three have begun to lose credibility as justifications for training. More executives and corporate training professionals are seeking quantitative evaluations, like ROI, that demonstrate value in monetary terms. Return on investment, or ROI, is a business term that means the ratio of money gained or lost on an investment relative to the amount of money invested. By conducting a training-focused ROI analysis, it is possible for training professionals to talk in the same financial language as other members of the management team. In addition, a quantitative analysis can increase trainers credibility and involvement in strategic decision making. In some cases, a positive ROI Quantifying the Organizational Benefits of Training analysis will result in the training function receiving more human and financial resources. Before embarking on an ROI analysis for a training course, organizations should consider what business goals the training is tied to. A second consideration is what baseline information is needed. In order to quantify the benefits that training has provided, it is necessary to compare performance to a pre-training baseline. It is essential that teams think about measuring ROI at the beginning of a training project. As teams start working on the quantification of training benefits, there are a few tips that can be useful. In general, it is a good idea to promote a culture that values measurement. Accountability should be incorporated into expectations and the reward system. The first time that a team conducts a training analysis, they should focus on a small scale project and make every effort to get the analysis right the first time. In this way, the training team will establish credibility with management. 2
Best Practices & Impact on Business The Origins of Training Evaluation & Measurement Training evaluation and measurement are not new concepts. In 1959, Donald Kirkpatrick (now Professor Emeritus at the University of Wisconsin) developed the Four Levels of Evaluation. Level 1, referred to as Reaction, measures how students responded to the training. Level 2 evaluates learning. More specifically, Level 2 evaluation assesses whether the participant advanced his or her knowledge with new skills and abilities. Level 3 is called Transfer. In this level of evaluation, trainers try to determine whether learners are applying the teachings from a training course successfully. The last level, Level 4, focuses on results more specifically what impact new skills have had on company performance. Level 4 is considered the most difficult type of evaluation to perform. With each level of evaluation, trainers gain a more precise measure of program effectiveness. However, each level also necessitates more time, focus, and analysis. The first three levels consist mostly of soft or qualitative measurements. Not surprisingly, most training departments conduct Level 1 evaluations with every course. Only about half of organizations use Level 2 evaluations. Level 3 and 4 evaluations are used even less frequently one quarter of organizations use Level 3 evaluations and only 12 to 13 percent of companies have taken on Level 4 evaluations. In recent years, the less quantitative levels (one through three) have begun to lose credibility as justifications for training. Many executives and corporate training professionals are seeking quantitative evaluations that demonstrate value in monetary terms. The Increased Interest in ROI & Training Return on investment, or ROI, is a business term that means the ratio of money gained or lost on an investment relative to the amount of money invested. There has been growing interest in applying ROI to the training function in organizations. This is due to three key trends. Before the recession, it was common for training budgets to grow each year. As company expenditures on training increased, the desire for accountability also grew, leading firms to apply ROI concepts to training programs. The second factor driving the interest in ROI and training is continuous process improvement. As organizations have embraced continuous process improvement for a wide variety of functions, the management staff has become more attuned to performance data and measurement. As a result, training departments have faced increased pressure to measure their output and successes. Finally, the popularity of outsourcing has caused many training professionals to focus on the bottom line. It is more common now to link training programs directly to business needs and to look for ways to improve training efficiencies. There appears to be a disconnect between what chief executives want in terms of training measurement and what chief learning officers are delivering. A recent survey of CEOs found that 96 percent of CEOs wanted to see learning and development connected to business impact data, but only 8 percent receive this information (Phillips, 2010). In addition, even though 74 percent of executives wanted to see ROI data, just four percent have access to it. The ROI Institute conducted a study which showed that only one in ten chief learning officers (CLOs) measured the impact of training on the business. Yet, nearly all CLOs measured success by how many employees were trained (Hill, 2009). Barriers to Quantifying Training Results One common barrier to quantifying training performance is the fear of disappointing results and the consequences, whether that takes the form of budget cuts, layoffs, or other penalties. In addition, many training departments have not invested adequately in measurement and evaluation processes. Most organizations spend one percent or 3
less of the learning and development budget on measurement. However, best practice organizations typically spend between three and five percent of the training budget on measurement and evaluation (Phillips, 2010). One common belief is that learning management systems (LMS) incorporate learning measurement. In reality, however, LMS tools rarely measure value or report on information that executives are interested in. Another issue is that training staff simply may not have the skills needed to implement an evaluation program. According to the 2006 Blanchard Survey, 58 percent of training and development professionals indicated that their biggest training implementation challenge is proving the impact and ROI of training ( Beyond our Ken, 2007). How ROI Can Be Applied to Training Because there is so much diversity in the types of training programs that organizations offer, there is no one ROI formula that can be applied to corporate training. It is important to bear in mind that the ROI of every training event does not need to be analyzed. Some types of training simply do not warrant analysis (such as new employee training or retirement planning). In addition, ROI analysis can be expensive to undertake. There are certain types of training programs that are well suited for ROI analyses. For example, those courses that are strategic to the business plan are good candidates, as are programs where management has high expectations that training will positively impact the company s bottom line. Training that is offered to large numbers of employees is also a good choice for ROI analysis. By conducting a training-focused ROI analysis, it is possible for training professionals to talk in the same financial language as other members of the management team. In addition, a quantitative analysis can increase trainers credibility and involvement in strategic decision making. In some cases, a positive ROI analysis will result in the training function receiving more human and financial resources. While ROI is a useful tool for quantifying training benefits, it does have some shortcomings. It does not do a good job at identifying how engaged learners were during a training course, nor is it effective at measuring how much information employees retain after training. Conducting an ROI Analysis for Training Before embarking on an ROI analysis for a training course, organizations should consider what business goals the training is tied to. Unless this linkage is determined upfront, data collection will be very difficult. A second consideration is what baseline information is needed. In order to quantify the benefits that training has provided, it is necessary to compare performance to a pre-training baseline. In the event that baseline information is not available, an estimate of the baseline can be made based on subject matter expertise, experience, or interviews. With the information that has been gathered, the team should design measures that will show whether or not the training accomplished its goals. It is essential that teams think about measuring ROI at the beginning of a training project. Effective ROI analysis processes share a number of different characteristics. They are simple, credible, theoretically sound, accountable for factors other than training which may affect performance, appropriate for use with many different HR programs, and usable with all types of data. As the team develops metrics for measuring the ROI of a training program, all the key stakeholders should be involved. When expectations are aligned with metrics, then valid measurements can be made. Conducting an ROI analysis is a multi-step process. 1. Identify the competency that most directly relates to the key business driver. After selecting a training program for analysis, it is important to identify the variables that are most closely related to accomplishing the business goals for the training. In this way, the focus of the analysis is narrowed to what matters most. 4
2. Link the competency to one or two key metrics. Determine how the key competency or skill taught in the training can be measured. Ideally, the metrics selected will have supporting data that can be gathered and benchmarked with other organizations. Examples of metrics include employee turnover, customer retention, increased sales, and reduced costs. Dimensions that ROI can be measured include efficiency (how much faster employees can do their jobs), effectiveness (how much better employees do their jobs), and customer satisfaction (how customers perceptions of the organization changed). Cost avoidance (how much money is saved on travel or time in the classroom) is particularly relevant to e-learning. 3. Project the savings derived from the training. This information can be gathered in a number of different ways. One option is to create a questionnaire and distribute it to training participants and their supervisors. Other alternatives are to conduct interviews or analyze existing reports. 4. Calculate the costs of the training program. Incorporate costs for things such as the course design and development time, training delivery costs, travel, and participant salaries. One option is to use average costs wherever possible. This will save effort, while still providing a good approximation of the program costs. 5. Calculate the ROI. This can be accomplished by dividing program costs by the key metrics that were identified early in the process. There are many different ways to quantify training savings, in addition to those mentioned in step three above. For example, one approach is to use employee action plans. During the training program, the participants prepare an action plan that outlines how they will apply the skills they have learned to their jobs. After the course, the employees share their plans with supervisors and any other stakeholders. A few months after the training, the participants and their managers meet and the employee reports on how they are progressing with the action plan. The managers develop a dollar value for the benefits generated by the training, and this information is provided to the training function. Another option is to use an accounting approach. While this is fairly straightforward for training costs, it can be more challenging for quantifying training benefits. Potential accounting measures for training benefits include faster work rates, less down time, fewer rejects, fewer lost sales, lower absenteeism, or fewer medical claims. When developing an ROI analysis for e-learning programs, hard and soft cost savings must be considered. Hard cost savings are usually easy to identify, such as eliminating travel to training sites or reducing the amount of time employees are away from work. Soft cost savings are more difficult to quantify, such as shorter learning curves for employees, improved information retention, or greater employee satisfaction. Other Measures for Training Effectiveness In addition to ROI, there are other measures that can be used to quantify training benefits. Two examples are impact on business (IOB) and return on expectations (ROE). Impact on business analyses focus heavily on performance. There are four basic steps required: 1. Understand the business goals. This stage requires comprehension of the goals, priorities, and metrics that internal customers value. This includes individuals at the executive level, as well as department heads. Before creating training programs, it is important to understand how employees are measured. 2. Re-engineer learning and development strategies. Given the information gathered in the first step, the chief learning officer and his or her team create learning strategies and objectives that are in alignment with the business goals. The training team may collaborate with department heads to identify the skills and knowledge that employees need to achieve the desired results. 5
3. Create tactics to deliver learning. Once the learning strategies have been defined, the training team will define programs that are engaging and maximize information retention. 4. Cultivate a continuous learning environment. IOB places an emphasis on continuous learning. The goal is that employees will continue to acquire knowledge and develop skills even after a training event is done. In a continuous learning environment, the training staff serves as mentors and facilitators, helping employees to apply new skills in the workplace. A second measure is return on expectations (ROE). With ROE, a brief interview (15 to 20 minutes) is conducted before the training with a key executive in the learning effort. Typically, this is a vice president who has financial accountability for the training. Based on the expectations expressed in the interview, a set of learning objectives are developed. After the training has concluded, the executive is interviewed once more and is asked to quantify the results of the training. This process tends to be less complex than ROI analysis, although the interviews require a skilled facilitator. Tips for Quantifying Training Benefits As teams start working on the quantification of training benefits, there are a few tips that can be useful. In general, it is a good idea to promote a culture that values measurement. Team members should be asked to think about results, accountability, measures, metrics, and analytics. Accountability should be incorporated into expectations and the reward system. It is also important to get executives more involved and to share successes with them. The first time that a team conducts a training analysis, they should focus on a small scale project and make every effort to get the analysis right the first time. In this way, the team will establish credibility with management. A number of techniques for evaluating training effectiveness are outlined: Follow-up assignments. After training is completed, participants are asked to complete a task which gives evidence that they understand and can apply the training content. This is a relatively easy technique to use and it is helpful when management support is not strong. Surveys and questionnaires. These tools capture student accomplishments and behavioral changes that occurred after training. Surveys tend to be inexpensive, as well as easy to administer and analyze. One-on-one interviews. Interviews with employees and managers are one approach for identifying changes in job related behaviors following training. Interviews are more versatile than questionnaires. Interviewers can explore issues and concerns related to training in more depth. Focus groups. Focus groups are designed to gather information from eight to twelve people in a structured setting. The group is asked specific questions about how training has changed the way they approach their work. In some cases, the process can initiate creative thinking that results in high-quality data. Performance contracts. Performance contracts are agreements that are made between employees, supervisors, and trainers before a training course begins. The group meets before the course to create measurable goals for the employee. After the training concludes, the group reconvenes to determine if the goals were met. Performance tracking. Performance tracking is one of the most credible post training evaluation techniques. The performance data of a department or individual is examined before the training, as well as after. This approach is often favored by senior managers. 6
Acronyms CLO: Chief Learning Officer IOB: Impact on Business LMS: Learning Management System ROE: Return on Expectations ROI: Return on Investment References Beyond our Ken. (2007, February). Training & Coaching Today, 4. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. ct=true&db=bth&an=24628867&site=ehost-live Gale, S. (2002). Measuring the ROI of e-learning. Workforce, 81(8), 74. Retrieved May 10, 2010, from EBSCO Online Database Academic Search Complete. http://search.ebscohost.com/login.asp x?direct=true&db=a9h&an=7164540&site=eh ost-live Gillette, B. (2002). Can you really measure ROI? Corporate Meetings & Incentives, 21(6), 32. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. http:// search.ebscohost.com/login.aspx?direct=true&db= bth&an=6974380&site=ehost-live Gillette, B. (2002). Pfizer s formula for measuring training s ROI. Medical Meetings, 29(6), 68. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. http:// search.ebscohost.com/login.aspx?direct=true&db= bth&an=8892171&site=ehost-live Goldwasser, D. (2001). Beyond ROI. Training, 38(1), 82. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. http:// search.ebscohost.com/login.aspx?direct=true&db= bth&an=4000836&site=ehost-live Grayson, C., & Roussos, L. (2003, July 28). Calculating ROI for traditional training. Point for Credit Union Research & Advice. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. http://search.ebscohost.com/ login.aspx?direct=true&db=bth&an=10381597& site=ehost-live Hicks, S. (2000). Why ROI? Training & Development, 54(7), 59. Retrieved May 10, 2010, from EBSCO Online Database Academic Search Complete. ct=true&db=a9h&an=3362267&site=ehost-live Hill, T. (2009). Measure smart: Trade ROI for IOB. Chief Learning Officer, 8(8), 28-31. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. ct=true&db=bth&an=43 581872&site=ehost-live How one trainer developed an eight-step program for ROI. (2005). HR Focus, 82(3), 10-13. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. ct=true&db=bth&an=16 285096&site=ehost-live Miller, D., & Mattick, B. (2006). The effect of training on the bottom line. Performance Improvement, 45(5), 13-18. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. ct=true&db=bth&an=24944793&site=ehost-live Moore, C. (2009). It s time for measurement strategy. Chief Learning Officer, 8(11), 40-43. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. ct=true&db=bth&an=45 433788&site=ehost-live Parry, S. (1996). Measuring training s ROI. Training & Development, 50(5), 72. Retrieved May 10, 2010, from EBSCO Online Database Academic Search Complete. http://search.ebscohost.com/ login.aspx?direct=true&db=a9h&an=960519147 5&site=ehost-live Phillips, J. (1996). ROI: The search for best practices. Training & Development, 50(2), 42. Retrieved May 10, 2010, from EBSCO Online Database Academic Search Complete. http:// search.ebscohost.com/login.aspx?direct=true&db= a9h&an=9602283405&site=ehost-live Phillips, J., & Phillips, P. (2010). Confronting CEO expectations about the value of learning. T+D, 64(1), 52-57. Retrieved May 10, 2010, from EBSCO Online Database Academic Search Complete. ct=true&db=a9h&an=47357253&site=ehost-live Prager, H., & Vece, S. (2009). Simplified ROI: Measuring what matters most. Chief Learning Officer, 8(11), 28-31. Retrieved May 10, 2010, from EBSCO Online Database Business Source Complete. ct=true&db=bth&an=45433785&site=ehost-live 7
Purcell, A. (2000). 20/20 ROI. Training & Development, 54(7), 28. Retrieved May 10, 2010, from EBSCO Online Database Academic Search Complete. ct=true&db=a9h&an=3362249&site=ehost-live 8