Survey Results: Delivering Virtual Instructor-Led Training (VILT)

Transcription

1 Survey Results: Delivering Virtual Instructor-Led Training (VILT)

2 Table of Contents Survey Overview 3 Key Findings 3 About the Study 4 Current State of VILT 7 Strengths and Limitations of VILT 14 Elements of Effectiveness 17 About General Physics Corporation 21 About Training Industry, Inc. 21 About This Research General Physics Corporation (GP) and Training Industry, Inc. 2

3 Survey Overview Key Findings Virtual instructor-led training, or VILT*, is quickly growing in popularity and use. The vast majority of respondents are increasing their use of VILT, and 33% or more are increasing their number of VILT courses by over 25%. However, 80% of respondents who use VILT did not think their companies are very effective at it. Their top two reasons are 1) difficulty engaging learners and 2) not fully using VILT tool functionalities. The study of 114 learning professionals estimates that companies use VILT in 27% of their courses. VILT courses are typically delivered with class sizes of 20 or less, with most respondents offering VILT courses for both customer/product training and professional development. To understand what may influence effective VILT use, study participants were divided into three groups by their effectiveness in using VILT. The very effective group uses more quizzes during VILT and chats and forums after VILT to engage their participants. They also tend to use more centralized support teams and leverage the experience of external providers more. On average, 27% of respondents courses are delivered using VILT, with about half using it as part of a blended solution with other delivery methods. The vast majority of companies are growing their use of VILT. The most common growth rate was over 25%. Thirty-three percent say their VILT courses grew by this much in 2009 and 37% expect this growth rate to continue in Only 20% of survey participants consider their companies to be very effective at using VILT. Engaging learners is a major challenge for VILT instructors. It is the greatest perceived limitation of VILT and the most often-cited difficulty among respondents who view their companies as not very effective at using VILT. Companies most often use VILT to reduce travel costs and extend training to more people (in more locations). Engagement tools are used more frequently during the VILT session than after the event. The very effective group uses more tools overall, including quizzes during VILT and chat and forums after. More of those in the very effective group use external providers and handle VILT functions through a central group. VILT is most commonly used for customers, professionals and sales staff. Most companies average VILT class size is 20 learners or less. *Note: For this study, VILT is defined as live or synchronous training in which instructors deliver courses through web, video and/or teleconferencing to remote attendees General Physics Corporation (GP) and Training Industry, Inc. 3

4 About the Study One hundred and fourteen learning professionals responded to this study in March Of those experienced in using VILT, some indicated they had been pioneer VILT users since 2001 or had used VILT for several years. On the other end of the spectrum were those in the beginning stages of experimenting with VILT. Survey questions focused on two areas: 1) Experience using VILT 2) Prospective plans, reasons for using and opinions about VILT Respondents represented a broad cross-section of companies which differed by industry, company size and VILT technologies used. Industries Please select the industry which best describes your company. Respondents came from a variety of different industries, with nearly half coming from three industries: Technology, Banking/Finance/Insurance and Business Services and Consulting. Refer to Figure 1. Figure 1 Technology (Hardware, Software, etc.) Banking/Finance/Insurance Business Services / Consulting Training and Development Telecommunications Health care Pharmaceuticals Non-profit Government Education (K - graduate) Construction/Engineering Services Transportation Manufacturing Wholesale Distribution Retail Other Respondents Industries 6% 5% 4% 4% 4% 4% 3% 3% 2% 2% 2% 7% 9% 17% 16% 15% N =109 Other= Utilities & Automotive Services 0% 5% 10% 15% 20% 2010 General Physics Corporation (GP) and Training Industry, Inc. 4

5 Employee Size Ranges Please select the employee size range which best describes your company. Respondents also came from companies of many sizes. Over half came from companies with over 1,000 employees, as shown in Figure 2. Figure 2 Respondents Employee Size Ranges 50,000 or more 9% 20,000-50,000 9% 10,001-20,000 6% 5,001-10,000 1,001-5, ,000 9% % % % 12% 15% 23% N =106 0% 5% 10% 15% 20% 25% VILT Technologies Which of the following VILT technologies has your company used? Check all that apply. The vast majority of companies use either WebEx or Live Meeting as their VILT platforms, as Figure 3 illustrates General Physics Corporation (GP) and Training Industry, Inc. 5

6 Figure 3 VILT Technologies Used WebEx (Cisco) 52% Live Meeting (Microsoft) 41% GoToWebinar (Citrix) Connect (Adobe) 23% 27% Internally developed product Centra (Saba) Sametime (IBM Lotus) ilinc N =101 11% 7% 5% 4% 0% 10% 20% 30% 40% 50% 60% The other technologies companies reported using include the following: Connect (AT&T) Elluminate Genesys Meeting Center Meeting Place Express (Cisco) Interwise ReadyTalk ReadyTech MegaMeeting.com Intercall Net Meeting Gatherplace Juniper SEE&SHARE Moodle Polycom video conferencing Proprietary LMS 2010 General Physics Corporation (GP) and Training Industry, Inc. 6

7 Current State of VILT To understand how companies are using VILT today, we asked a series of questions about how effective they are, how often they use it, which topics they teach and for how many learners, and what other delivery methods they blend with VILT. Effectiveness Ratings Overall, how effective has your company been in using VILT as a delivery tool? While VILT is in a high-growth period, many appear to be on a learning curve as they adapt course materials and instructional techniques to work optimally with the new technology. Among those using VILT, only one in five considers his/her company to be very effective at it, as Figure 4 shows. Over half view their companies to be somewhat effective at it, and over one in four believes his/her company is even less than somewhat effective together, these three responses form the less effective group. Figure 4 60% 50% 40% 30% 20% Company Effectiveness in Using VILT Group 3: Less Effective Group 2: Somewhat Effective 53% Group 1: Very Effective 20% 10% 7% 11% 9% 0% N =101 Very Ineffective Somewhat Ineffective Neither Ineffective or Effective Somewhat Effective Very Effective To better understand what VILT practices may impact effectiveness, it is useful to compare the VILT uses and practices of those in the very effective group compared to those in the other two groups. Notable differences between the groups will be highlighted throughout the report. 7

8 As expected, those in the very effective group use VILT in more courses than the other groups. In fact, they use it in about twice as many of their courses: 51% versus 26%.This can be interpreted in at least two ways: 1) more experience with VILT courses leads to greater effectiveness and 2) those who are more effective at VILT adopt VILT for more of their courses. Reasons for Effectiveness Ratings Why did you give this effectiveness rating? What metrics do you or would you use to measure effectiveness? Respondents gave two types of answers to these questions: A) how they measure VILT effectiveness and B) explanations for their ineffectiveness at using VILT. A) Measurement of Effectiveness Respondents reported four main ways in which they evaluated VILT effectiveness. They are listed below from the most to least common ways. Learner, Customer and Instructor Feedback and Satisfaction Rates (44%) By far, feedback after the course is the most common means of assessing VILT effectiveness. This includes end-of-course or Level I evaluations, or satisfaction and other rating surveys from learners in areas like content, delivery, facilitation and engagement. While most indicators of engagement seem to come from the learners themselves, instructor feedback can also include observations of the types and numbers of questions asked. Respondents typically compare feedback ratings to physical classroom instructorled training (ILT). Learning Outcomes (22%) This includes knowledge tests, scores from test series and a variety of assessments. Event Popularity/Participation Rates (19%) Some respondents also gauge VILT effectiveness by the numbers who enroll, attend and complete VILT courses. Metrics include monthly averages of VILT sessions, students taking VILT courses and minutes of technology utilization. Transfer of Learning to Work & Business Results (15%) This can include manager interviews of how learners are applying the training to their jobs, line productivity metrics and efforts to measure the return on investment by comparing the benefits of the VILT training to the costs. Some also measure cost savings as another outcome metric. Overall, the level of sophistication in measuring the effectiveness of VILT is still a challenge to many organizations. B) Explanations for Ineffectiveness Survey participants in the somewhat effective or less effective groups offered three main reasons for not being very effective. The most common reason they cited was difficulty 8

9 engaging learners and getting them to participate. Some struggled to maintain standards of interactivity with areas such as discussions, question and answer periods, collaborations and networking. Several respondents also observed that instructors were not fully utilizing VILT tool functionalities. They admitted to just scratching the surface of what the tools could do, falling into ruts in using limited parts of the tools and even trying to mimic the physical classroom rather than adapting their content to the new delivery method. Some specifically pointed out the need to learn how to use chat, video clips and polling. Finally, some survey participants said that technology issues hampered their effectiveness. They complained about technical issues such as slow or poor connectivity, low-quality graphics, sound or video, as well as restrictions on the network, such as the number who can access hands-on labs at one time. Growth For and , select the option that best describes your company s actual or expected change in the number of VILT courses offered. The use of VILT is on the rise. Altogether 84% grew their use of VILT last year and 89% expect to grow it in As Figure 5 shows, the greatest percentage of respondents, 33%, experienced growth rates of 25% or more in 2009; 37% expect this level of growth in Virtually no companies plan to decrease their use of VILT. Figure 5 Growth in Number of VILT Courses Increase of over 25% 33% 37% Increase of 15-25% Increase of % Increase of 5-9.9% Increase of less than 5% Stayed about the same 18% 18% 11% 14% 12% 11% 10% 9% 16% 10% Actual: 2008 to 2009 Expected: 2009 to 2010 Decrease 0% 2% N = 92 (Actual) N = 102 (Expected) 0% 10% 20% 30% 40% 9

10 Use Please specify the percent of all courses your company offered in 2009 using VILT, including the percent where VILT is used by itself and the percent where VILT is used as part of a blended solution. Respondents reported that they used VILT in an average of 27% of their courses, as Figure 6 illustrates. Of those VILT courses, about half are conducted with VILT being one part of a total blended solution. The other half is conducted exclusively with VILT by itself. Figure 6 Percent of All Courses Using VILT % of All Courses % of VILT Courses 53% Courses with VILT 27% Courses without VILT 47% By Itself As Part of Blended Solution N = 59 N = 78 Among respondents companies that use VILT to deliver any of their courses, 85% blend VILT for at least some of their courses. Other Delivery Methods When using VILT as part of a blended learning solution, which of the following does your company typically include? Check all that apply. The majority of respondents who blend VILT use three other delivery methods, as shown in Figure 7: self-study online or elearning, ILT and print/support materials. It is interesting to note that so many companies blend VILT with ILT which means that the VILT may be used for selected segments of a larger class with in-person sections. While archived sessions may have the advantage of further extending a prime benefit of VILT reaching more people (e.g., in different time zones or with schedules that cannot be met 10

11 during the live VILT sessions), only slightly more than one in three respondents selected that his/her company use this option along with live VILT. Figure 7 Delivery Methods Typically Blended with VILT Self-study online/elearning In -person instructor -led training (ILT) Print/other support materials Testing Archived sessions Chat rooms/online forums Wiki's Simulations, games & virtual worlds Podcasts 40% 36% 26% 24% 21% 13% 74% 68% 67% N = 87 0% 10%20%30%40%50%60%70% 80% Less than one in three respondents use some of the emerging social, on-demand and other learning technologies such as chats/forums, wiki s, simulations/games and podcasts. Still other respondents added that they combined VILT with Computer-Based Training (e.g., with CD- ROMs) and team or peer-led meetings outside the course. Class Size What is your company's average number of participants in a VILT course? Seventy-five percent of companies have average class sizes under 20 learners, as Figure 8 shows. This class size is similar to that of many ILT classes. 11

12 Figure 8 25% Average VILT Class Sizes Similar to ILT Class Sizes 20% 15% 16% 16% 20% 15% 10% 5% 7% 5% 10% 3% 7% 0% N = 91 3 to 5 6 to 8 9 to to to to 25 Class Size Ranges 26 to to to 500 Although one might think that the very effective group, which presumably does a better job of engaging students, might have a smaller class size to better engage with learners, the opposite is true. The very effective group has a significantly higher average class size than the other two groups. This may mean that the very effective group gained confidence in how to engage learners in smaller class sizes and then expanded their use of VILT to larger audiences. Topic Areas For which of the following topic areas has your company delivered training using VILT? Select all that apply. Most respondents companies use VILT for customer/product training as well as professional development, as shown in Figure 9. Many also use VILT for sales training. To varying degrees, all three types of training share the common need to be delivered to learners in different locations. 12

13 Figure 9 Topics Delivered using VILT Customer/product training 58% Professional development 54% Sales training 42% Compliance training 33% Leadership training 33% IT 32% Management/supervisory training 29% Onboarding 29% Customer service 26% Administrative topics 25% Project Management 22% N =101 0% 10% 20% 30% 40% 50% 60% 70% 13

14 Strengths and Limitations of VILT Why are some companies driven to use VILT, and what makes others hesitate before they convert a course to VILT? Further, to become an effective VILT instructor, what types of skills are needed to get up to speed with the unique requirements of VILT? Drivers Since VILT is growing at such a strong rate, respondents must believe that the perceived benefits (or drivers) outweigh the perceived limitations. But what are the drivers that are propelling the growth of VILT? Rate how important each of the following was in driving your company to use VILT. As Figure 10 clearly shows, the most critical driver for using VILT is the need to reduce both travel costs and time. Further, 46% rated this critical, with a combined 90% rating it either critical or important (i.e., importance ratings). Particularly when training organizations are asked to do or train more with less, the motivation to keep travel costs down can be compelling. The second most critical driver is increasing the number of people trained. Some respondents further explained that they specifically need to reach more people in dispersed physical locations. Figure 10 Importance Ratings for VILT Drivers Reduce travel costs and time 46% 44% Increase number of people trained 28% 47% Increase effectiveness 14% 45% Deliver training with greater consistency 18% 39% Improved VILT technology available Increased user acceptance for VILT 9% 8% 47% 46% Critical Important Reduce time in physical classrooms 18% 34% Involve more co. leaders/subject experts 11% 28% Measure attentiveness and participation 5% 25% N =106 0% 20% 40% 60% 80% 100% Of about equal importance are a few drivers in the middle: delivering training with greater effectiveness and consistency, and leveraging the availability and acceptance of VILT technologies. 14

15 Limitations What limitations do you see for using VILT? Select all that apply. By far, the majority of survey respondents felt that lack of full engagement was the greatest limitation of VILT, as shown in Figure 11. Technology limitations of all kinds were the secondhighest limitation to using VILT, reflecting both difficulty in using technology that is not easy for all to use and the still-developing stage of these technologies. Note that the limitations of VILT overlap heavily with the explanations for ineffectiveness listed above especially lack of engagement. Clearly engaging learners is a major challenge for VILT, but one that the very effective group may have overcome. Figure 11 Limitations of VILT * Lack of full engagement 71% Technology 43% Cost Finding qualified instructors High cancellation rates 23% 20% 18% * Also #1 reason why companies are not effective at using VILT Not meeting attendance goals 14% N =109 0% 10% 20% 30% 40% 50% 60% 70% 80% Instructor Skills How important is it that VILT instructors have each of the following? As with ILT, respondents believe it is important that VILT instructors also have facilitation skills and topic matter expertise. The two additional skills that they also expect VILT instructors to have are an ease in using technology in general and VILT-specific skills, as Figure 12 shows. 15

16 Figure 12 Importance of VILT Instructor Skills Facilitation skills 61% 35% Topic matter expertise Ease with using technology in general Overall VILT-specific skills training Understanding of audience needs 59% 32% 29% 38% 53% 53% 40% 35% Technology skills in general and with VILT in particular are also important. Classroom management skills Training material devel. capabilities Instructional design capabilities 23% 11% 32% 9% 33% 52% Critical Important N =110 Ability to assess ROI of VILT 6% 31% 0% 20% 40% 60% 80% 100% The fact that one of the main explanations for ineffective VILT use was that instructors were not fully using the VILT functionalities further reinforces the importance of conducting VILT-specific training and having an ease with using technology in general. Other skills respondents wrote in as being critical were communication, listening and audiencereading skills which could all be considered facilitation skills as well. They also valued adaptability (e.g., being flexible when things change) and the ability to follow up (e.g., with managers) after the training session. 16

17 Elements of Effectiveness Now that we understand how companies are generally using VILT and what its perceived strengths and limitations are, we take a closer look at three elements that study respondents associated with the effectiveness of their VILT programs: 1) what drove them to use it; 2) their use of engagement tools; and 3) the way they source and support it. Drivers: By Effectiveness Groups Rate how important each of the following was in driving your company to use VILT. When the drivers for using VILT were analyzed for each of the different effectiveness groups, differences were found for six of the nine reasons. Figure 13 highlights these differences, starting with the strongest differences (about 40%) in importance ratings between the very effective and less effective groups for the first three reasons: using VILT to improve the effectiveness of their training and the increased availability and user acceptance of new VILT technologies. Figure 13 Importance Ratings for VILT Drivers by Effectiveness Groups Reduce travel costs/time Measure attentiveness & participation Deliver training with greater consistency Increase the number of people trained Reduce time in physical classrooms Involve more leaders & subject experts 40% 22% 28% 48% 56%65% 43% 35% 45% 44% 60% 59% 94% 100% 81% 72% 80% 78% Strongest difference between groups (critical + important) Very Effective Somewhat Effective Less Effective N = 106 0% 20% 40% 60% 80%100% 17

18 What could these differences mean? Those who are very effective are more seasoned users with a higher percentage of their courses using VILT; they may have found more important reasons for using VILT as they expanded their use of VILT over time, in addition to their reasons for initially adopting VILT. Once they become competent at using VILT, their focus may shift to increasing the effectiveness of their training using VILT. They may also want to convert more courses to VILT due to demands from learners who have become comfortable with VILT for other courses or to take advantage of new technologies with enhanced features and benefits. Engagement Tools Which tools do you use to keep your audience engaged during and after VILT? As Figure 14 shows, far more respondents use engagement tools during VILT sessions than after. Specifically, the majority use live conversations, chat, poll and quizzes during VILT. After VILT, the preferred tools include quizzes, live conversation, blogs, forums, chat and polls. Figure 14 Live conversation Chat Polls Quizzes Animation Rapid advancement Forums Blogs Twitter LinkedIn Facebook Use of Engagement Tools for VILT During 8% 3% 2% 1% 1% 82% 64% 54% 54% 48% 38% Quizzes Live conversation Blogs Forums Chat Polls Animation Twitter LinkedIn Facebook Rapid advancement After 31% 24% 23% 19% 15% 13% 8% 8% 8% 6% 5% N = 111 0% 50% 100% 0% 10% 20% 30% 40% The very effective group uses quizzes during VILT and chat and forums after VILT more often than the other two groups. In addition, the very effective group uses slightly more tools during VILT (an average of 4.2 tools versus 3.9 for the other two groups) and more tools after VILT (an average of 2.4 tools versus only 1.6 for the other two groups). In addition to the tools listed above, respondents also said that they used breakout rooms, whiteboards, remote labs, their company s own social media tools, sharing tools/sites and wiki s to engage their learners. 18

19 Sourcing and Support How does your company support each of the following VILT processes? Respondents companies are fairly evenly split between handling many VILT support functions internally by one group (i.e., centralized) or by multiple groups (i.e., decentralized), as displayed in Figure 15. Only one in ten or less share the responsibility of VILT functions between internal resources and external service providers. The back office functions of scheduling or administration and evaluation make the least use of external service providers and are the most centralized areas. Figure 15 Sourcing and Support of VILT Functions Course delivery 10% 45% 43% Event facilitation 7% 42% 47% Evaluation 6% 36% 55% Both internally and externally Internally by multiple groups Internally by one group Scheduling or Admin 5% 35% 59% N = 97 0% 20% 40% 60% 80% 100% Sourcing and Support: By Effectiveness Groups When the sourcing of VILT functions are charted by effectiveness groups, a clear and consistent pattern emerges. As figures 16 and 17 show, the very effective group centralizes all VILT functions and uses external providers along with internal staff to support VILT functions more often than the other two groups. By centralizing and using external service providers for VILT functions, companies can take advantage of VILT expertise both in-house and among service providers who have the added perspective that comes from assisting many companies in how to best use VILT. 19

20 Figure 16 80% 70% 60% 50% 40% 62% 53% Centralization of VILT Functions by Effectiveness Groups 70% 60% 48% 65% 55% 49% 50% 43% 42% 42% 30% 20% 10% Less Effective Somewhat Effective Very Effective 0% N = 97 Scheduling or Admin Evaluation Event Facilitation Course Delivery Figure 17 16% 14% 12% 10% 8% 6% 4% 2% Sourcing VILT both Internally & Externally by Effectiveness Groups 10% 8% 8% 2% 2% 15% 8% 10% 4% 4% 12% 15% Less Effective Somewhat Effective Very Effective 0% Scheduling or Admin N = 97 Evaluation Event Facilitation Course Delivery Delivery and evaluation are the two areas where companies most often used both external providers and internal staff to support their VILT courses. 20

21 About GP General Physics Corporation (GP) is a global training, consulting and engineering company that helps performance-driven organizations solve business issues while creating pathways for continuous improvement. GP s approach to teamwork is personalized and reinforced by a strong commitment to earning client satisfaction. GP s professionals are dedicated to providing the superior service and flexible solutions that have been the hallmark of the company for more than 40 years. From custom training, sales training and consulting to talent management and business process outsourcing, when working with GP, clients can count on a tailored approach that focuses on their business goals. For more information, visit or call About Training Industry, Inc. Training Industry, Inc. and our portal, TrainingIndustry.com, spotlight the latest news, articles, case studies and best practices within the training industry. Our focus is on helping dedicated business and training professionals get the information, insight and tools needed to more effectively manage the business of learning. For more information, go to or call About This Research Copyright 2010 by GP and Training Industry, Inc. All rights reserved. No materials from this study can be duplicated, copied, re-published or re-used without written permission from GP or Training Industry, Inc. The information and insights contained in this report reflect the research and observations of GP and Training Industry, Inc. analysts. 21