Training Analytics Market Analysis:

Transcription

1 Training Analytics Market Analysis: How Do Companies Measure Training and ROI? Bersin & Associates Bersin & Associates March, 2005 Bersin & Associates and LTI Magazine 1 August, 2003

2 Training Analytics: A Market Analysis Date of survey: October, 2004 Surveys sent: 5,999 Respondent 67% Corporate Profile: 18% Education, 15% Government In October of 2004, Bersin & Associates conducted an on-line survey of nearly 6,000 subscribers to understand how they are currently measuring the effectiveness of their training programs. Why did we choose this topic? We are keenly interested in measurement practices because we believe that measurement, analytics, and benchmarking are critical tools to help training organizations optimize their programs to drive higher and higher business value. In general what we found is that companies are keenly interested in measuring business impact from training (Kirkpatrick Level 3 and 4) yet struggle with a clear lack of methodologies and tools. Most companies spend 1%-3% of their overall training budget on measurement but have a strong interest in spending more. LMS systems, while often purchased for the purpose of measuring training operations, are falling short in providing this capability. Training Analytics Survey: Key Findings 1. Most organizations routinely measure training volumes (enrollments, completions, hours delivered), Learner satisfaction, and total training costs. These measures are the easiest to obtain and are widely used. 2. The majority of organizations consider the business impact of training to be the most valuable information to obtain, but very few firms are actually measuring it. The reason for this is a clear lack of tools and methodologies. 3. The top business impacts which companies measure are sales-related, qualityrelated, retention, and overall cost reduction. 4. Most organizations are not satisfied with their Learning Management Systems ability to provide the requisite measurements and reporting information. 5. Spending on tracking and measurement is small today, but there is a strong demand for higher spending and resource allocation in this area. What do Companies Measure? Bersin & Associates 2 March, 2005

3 The first question we asked was what areas do companies measure today? As the chart shows below, training volumes (enrollments, completions, and student hours) are the top measure, followed closely by satisfaction. This shows that companies still benchmark their operation by How much training did we do and how well did our audience like our programs. Unfortunately, these are not business-related metrics nor can they be used to rank business impact across training programs. When asked about measuring costs (an Efficiency measure), only about half of companies routinely measure the cost of individual programs, and only about a quarter actually compare costs between programs. Again we see an opportunity for improvement here. Cost comparisons across programs can be valuable information in scoring program efficiency and ROI. Companies that do not measure costs are clearly not measuring ROI. When asked about measuring true business impacts, the numbers are even lower. Only 14% have any regular way of measuring job impact (Level 3) and 11% measure business impact (Level 4). There are several reasons for this. First, respondents clearly told us that they are lacking tools and methodologies to perform these correlations. Second, many organizations are not given the budget or resources to perform these types of measurement projects. Information Routinely Tracked Business Impact Job Impact Cost comparisons Cost per student/hr. Student Scores Certifications/Compliance Total cost per program Total hrs. delivered Completions Learner Satisfaction Enrollments 11% 14% 28% 39% 50% 52% 57% 74% 76% 76% 82% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Figure 1: What information do companies track today? (All organizations) Our Measurement Taxonomy In our research in this area we have created a taxonomy or way of characterizing all the possible things that can be measured. We group all measurements into three categories: Bersin & Associates 3 March, 2005

4 1. Effectiveness. How well does this program actually impact learning, job results, business results, or cost savings. 2. Efficiency. How cost-effectively did we reach the audience? What was our cost per student or student-hour and how does it compare with other programs? Effectiveness Efficiency Compliance 3. Compliance. Did we achieve our compliance targets? Who is in compliance and how is not? What level of compliance did we achieve? What is our risk of falling out of compliance? You can read more about our measurement approaches in our whitepaper, Training Analytics, the Time is Now. Learning Results: Scores and Certifications A critical goal of training is to improve human performance; hence one would expect a very high percentage of firms would measure scores or other learning-related metrics. This was not true. In our survey only about half of the respondents stated that they routinely track scores for training programs. Our qualitative surveys show that sores are used and tracked when the program has a mandatory certification nature. Why is this? If you go to the trouble of training people, shouldn t you always measure whether or not they learned anything? There are many reasons why programs are not scored: lack of infrastructure, the program did not have the budget, inability to decide what to measure, or just lack of time. These are not excuses but rather real business limitations. We would hope this number to go higher over time as more and more technology is applied to training, but as described above, many training departments do not have the budget or focus on this area. There is also some confusion about whether assessments can adequately measure actual learning and job performance. One participant commented, Pre-testing has come and gone because we are not clear that is best use of time and that it is accurate representation of learning. We always have some degree of post-testing - as an effort to reinforce key messages rather than to measure learning. This comment indicates that scoring is more of a motivational tool than a measurement tool. Another aspect to scoring is the need to measure workers on-the-job. One participant described his company s technique in this way: We regularly track observed performance of skills. We are a high tech company and train technicians to fix and repair our products. Physical skills are observed prior to certification. As any experienced training professional knows, demonstration of on-the-job skills is the most valuable way to truly test Level 2 learning. Compliance or Certification Driven Programs Many training programs are compliance or certification driven. 52% of respondents told us that they regularly run these types of programs. We define these as programs that must be strictly monitored for completion, where completion may be defined as simply enrolling, actually completed, or passing a series of assessments. These programs are often driven by actual learning needs or may be driven by government regulation needs. Bersin & Associates 4 March, 2005

5 In some cases learning is not the driver, hence scores are not used. A good example is sexual harassment training. If a company certifies that all its managers have completed sexual harassment training, they can no-longer be sued for certain types of sexual harassment claims. The score or learning results of this program are not relevant to the business need. The business driver is simply to make sure that everyone has completed. Measuring E-Learning vs. Traditional Approaches With the surge in e-learning over recent years, a number of firms are evaluating their e- learning programs relative to their classroom-based programs. One participant commented, We track why students elect to take web-based training when classroom training is available, the hours when students want to take web-based training, if the costs of the web-based training were reasonable, and what element (trainer, presentation, graphics, etc.) of the training was the most and least valuable. Analysis: This is a very healthy trend. In the early days of e-learning companies were just happy to get courses online. We have reached the maturity level now where organizations are starting to try to benchmark the effectiveness of their e-learning vs. other approaches, and often find that e-learning does fall short. As our Blended Learning: What Works study found, companies are quickly finding that for certain high-impact programs, blended delivery is needed. What Job Impact do Companies Want to Measure? Very few firms, less than 15%, are measuring learners behavioral changes or the job impact of training programs, and only one in ten measures the business impact of their training programs on a routine basis. For those that do go to this level, what job impacts do they measure? Analysis: This problem is one of the big reasons why we are very positive and working hard to help companies understand the value of Training Analytics systems, which make this problem much easier to solve. When asked what types of business measures do you try to correlate to training, the list was as follows: Business Measures Companies Want to Correlate to Training Measure % of Organizations who want to use It Employee Turnover 11% Employee Productivity 19% Bersin & Associates 5 March, 2005

6 Sales Revenue 16% Sales Cycle Time (time to close a sale) 10% Time to Market 13% Product or Service Quality 29% Operational Cost Savings 24% Figure 2: What business measures do companies want to correlate? We were surprised by the number of comments from companies who want to measure other business impacts. Some other areas include: Quality of service done the first time (rework) Service call surveys (customer satisfaction) Accidents (worker compensation incidents) Increased sales of trained VARs vs. untrained VARs (value-added resellers) Speed of answer (technical support). Cost Savings Frequently Used as a Measure As cited previously, costs are a primary concern and the most common way of assessing training s impact on the business. When asked about specific data being measured, cost savings was most often cited. One participant explained his company s measurement activities: I do an annual survey to determine Level 3 (Job impact of training). In the same survey, I ask managers/employees to tell me how the program has saved time/cost and to be specific in terms of dollar amount. The Trap of Cost Reduction (Commentary) The most popular measure of business performance we found (and continuously hear) is cost reduction. This particular program reduced costs by xyz. If these cost reductions are actual cost reductions in business operations, these are excellent measures. However, if companies are trying to score their programs by the amount of money saved on training, this becomes a trap. Consider this. Most companies spend 1-3% of their payroll on training. If the training organization saves 20 or 30% of this, the overall company savings will be a very small fraction of overall company expenses. There is usually not that much budget in the training organization to save. Moreover, most e-learning savings are a one-time effect. The first year things look cheaper, but what do you do for the second year? Ultimately if you really want to save money on training, you can eliminate it altogether. We think people should seriously consider whether this is a valid measure at all. The business impact of training is X higher when viewed as a performance enhancer, so if you save $10 in training delivery but potentially lose $ in company performance, the savings are clearly nil. Bersin & Associates 6 March, 2005

7 Smaller Companies More Easily Measure Impact Small to medium-sized organizations (those with less than 1,000 employees) were somewhat more likely than large enterprises to measure the job or business impact of training. Seventeen percent (17%) of Small to medium-sized organizations reported that they routinely measure the job and business impact of their training programs. Just eleven percent (11%) of large enterprises measures the job impact and only 5% measure the business impact of training. We believe the reason for this is that smaller companies just have an easier time obtaining data and talking with managers about what impact a training intervention had. Large enterprises find it difficult to obtain good data due to the complexity of their internal structures and systems, as shown below. Some participants believe it is too difficult to assess the business impact of training. As one individual commented: The only relevant measures are employee-specific. More general business measures are confounded by too many other factors; it is and would be impossible to identify the training impact, per se, except in the broadest overall way. A Clear Lack of Tools and Processes We asked people what they wanted to measure more closely and what issues they face that holds them back. The greatest need companies have is to understand how to better measure business impact. Why do so few do it (less than 11%)? Companies clearly lack the tools and processes to make such measurement easy. 70% of respondents said that they would measure the business impact of training programs more routinely if they had better tools, and the same number said they would be helped by having more formal processes or methodologies in place. Needs for Measuring Business Impact of Training Better process or methodology 71% Better measurement tools 70% Higher priority by mgmt. 63% More resources or budget Better reporting system 61% 63% All Organizations 0% 10% 20% 30% 40% 50% 60% 70% 80% Figure 3: What do companies need to improve measurement processes and results? Large enterprises were more likely than small to medium-sized organizations to cite a lack of tools and processes in preventing them from measuring the business impact of their training programs. We believe this is because small companies can more easily talk to managers and obtain information on their audiences. Bersin & Associates 7 March, 2005

8 LMS Systems Fall Short We regularly study the Learning Management Systems (LMS) market, and recently identified that centralizing information about all my training investments is now the #1 reason companies give for purchasing an LMS. Despite this need, the promise appears not to be met. While large enterprises are more likely to have an LMS in place, of the companies who had an LMS, only 3% gave their systems top marks for being able to provide the needed measurements and reporting. In fact, 22% rated their LMS poor in providing reporting and measurement, and 39% rated their LMS fair. Our findings say that after going to the expense of purchasing an LMS, only about a third of companies are satisfied with the reporting and measurement solution they have obtained. There could be many reasons for this and our whitepaper Training Analytics: The Time is Now describes some of these in detail. We believe that this is an important new area for LMS systems, and strongly recommend that companies learn about Training Analytics before selecting or purchasing an LMS system. Rating of LMS's Ability to Provide Necessary Measurements & Reporting Poor 22% Fair 39% Good 36% Excellent 3% All organizations 0% 10% 20% 30% 40% 50% 60% 70% 80% Figure 4: How would you rate your LMS in its ability to provide you the measurement and reporting you need? Bersin & Associates 8 March, 2005

9 Demand for Greater Spending Although measuring the impact of training is considered to be very valuable by most firms, very few have put significant resources behind it. The majority (51%) of participants who were knowledgeable of their departmental spending said they spend less than 2% of their total annual training budget on tracking and measurement. Large enterprises spend more than smaller organizations, but the spending is still extremely limited. Spending on Measurement as % of total training budget less than 2% 44% 57% 2-5% 22% 33% 5-9% 13% 18% Small/Medium-Size Organizations 10% or more 4% 9% Large enterprises 0% 10% 20% 30% 40% 50% 60% Figure 5: What percentage of your training budget do you spend on measurement and analytics today? Most participants recognize that more needs to be spent on tracking and measurement. One in four said they should be spending much more on tracking and measurement activities and half said they should spend slightly more. We see a latent demand for better analytics solutions. Conclusions and Analysis: Opportunity for Impact The subject of training measurement is one filled with research, opinions, methodologies, and case studies. Despite this body of knowledge, we find over and over that our clients and research participants struggle with the lack of tools and methodologies to measure, benchmark, and score their training investments. Hence in most organizations, there is a tremendous opportunity to more effectively measure training. As you have probably read, If you can t measure it, you can t manage it. We are very enthusiastic that companies understand the problems they face and are working hard to improve this critical part of the training organization. Bersin & Associates 9 March, 2005

10 Additional Resources Available We have invested a lot of our research into this market, and have many resources to help companies further understand the why s, how s, and what s of training measurements and analytics. These include our whitepaper Training Analytics, The Time is Now and our recent 3-part webinar series on Training Analytics. Both are available on our website by going to If you become a research subscriber, you can receive additional methodologies and detailed case studies on best practices. About Us Bersin & Associates is a leading provider of corporate and vendor consulting services in enterprise learning technology and implementation. With more than 20 years of experience in e-learning, training, and enterprise technology, Bersin & Associates provides a wide range of services including product development, product marketing, industry research, corporate workshops, corporate implementation plans, and sales and marketing programs. Some of Bersin & Associates innovations include a complete methodology for LMS selection and application usage, an end-to-end architecture and solution for e-learning analytics, and one of the industry s largest research studies on blended learning implementations. Bersin & Associates can be reached at or at (510) Bersin & Associates 10 March, 2005