Lynette Gillis, Ph.D. and Allan Bailey, Centre for Learning Impact

Transcription

1 CASE STUDY DILLON CONSULTING: Measuring the ROI of Project Management Training Lynette Gillis, Ph.D. and Allan Bailey, Centre for Learning Impact Overview of Dillon Consulting Dillon Consulting is a Canadian-owned professional services firm operating across Canada and internationally. Dillon manages and carries out consulting projects in wide ranging business domains planning & development, environment, health & safety, municipal engineering, facilities construction, and transportation engineering. Dillon s 600 employees include professional engineers, architects, planners, economists, and physical and social scientists. A guiding ambition within management is to maintain a level of skill and professionalism that enables the firm to deliver to clients the highest value in terms of quality, service, timeliness, and fees. Adhering to this goal has contributed to the quadrupling of profits over the past four years and the firm s selection by the Financial Post in February 2009 as one of Canada s 50 Best Managed companies for the third consecutive year. Training Rationale A key element of this success has been to hire and nurture the highest quality professionals. In 2007, Dillon decided to invest in the development of a new training initiative designed to improve a variety of key performance objectives for project managers including enhancing knowledge and skills in project planning, budgeting, managing quality and risk, and negotiation, etc. The primary business objectives of the training, PM 201, were to improve two key business metrics: Total Collection Period (TCP) and individual Project Manager s Profitability. The Training PROJECT MANAGEMENT 201 Target Audience: All project managers who had previously completed the company s introductory project management course Format: Two-day, face-to-face workshop, conducted off-site, in different regions of the country Content: Overview to Company Vision, Mission, Culture and Management Philosophy; Getting the Project; Planning the Project, Executing the Project, and Closing the Project; Leadership and Motivating Teams Activities: Pre-Work, Self-Assessments, Team Activities, Group Discussion, Case Study, Simulation, Video, Small Group or Individual Reflections, Action Planning AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 1

2 The Case Study Methodology THE LEARNING VALUE CHAIN The Investing in People (IIP) case studies adopt the Gillis and Bailey concept of the Learning Value Chain as a general framework for evaluating the learning effectiveness and the business value of training and human resource investments (Figure 1). In the Learning Value Chain, the training program triggers a chain of critical outcomes. As desired outcomes are achieved at each link along the chain, greater value is added and the likelihood increases that training will result in positive business outcomes and return on investment. Conversely, if training fails to meet outcomes at any link, value is diminished and the prospect of positive business results and return on investment is at risk. Using the Learning Value Chain Methodology, the training program is evaluated at each of four links (Capability, Transfer, Business Results and ROI). At each link, data is gathered to assess the extent to which the training has achieved key outcomes, added value and enabled the next critical event in the chain to occur. The Learning Value Chainmodel also incorporates a diagnostic strategy to investigate training practices and strategies that may strengthen or weaken outcomes at each link and subsequently influence business impact and return on investment. See Appendix A for a detailed description of the methodology used in the Investing in People studies. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 2

3 Figure 1 The Learning Value Chain Training Activates a Chain of Critical Outcomes CAPABILITY Training Enhances Capability and Intentionality: As a result of training, participants acquire new knowledge, skills or attitudes enhancing their capability to take on new work roles and tasks or improving the way they currently do their work. They also develop the intention to apply their learning and begin to formulate plans or ideas for doing so. TRANSFER Enhanced Capability Improves Job Performance: The work context and learning strategy supports the transfer of learning to the workplace. As a result, participants successfully demonstrate new behaviours, perform new roles/tasks, or improve their job performance. BUSINESS RESULTS Enhanced Job Performance Improves Business Results: Changes or improvements in the way in which participants perform their work contribute to positive improvements in closely linked business or organizational outcomes. ROI Positive ROI: Net benefits from business improvements exceed training costs, resulting in a positive return-on-investment (ROI). AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 3

4 CAPABILITY RESULTS Evaluation Questions: Did training participants develop new capability: acquire new knowledge and skills, adopt new attitudes, improve existing skills, or discover new and more productive ways of doing work? Did participants also develop the intention to apply their learning or improve their job performance? PM 201 succeeds in enhancing Capability To assess Capability the first link in the Learning Value Chain all training participants completed the Capability Questionnaire at the end of the workshop. The results suggest that PM 201 succeeds at the first link in the Learning Value Chain. Participants perceive large gains in knowledge and skill across key learning domains as a result of attending training. Moreover, they give strong indication that they are able and motivated to apply their learning to their jobs. The questionnaire results are summarized in the Capability Index (at right). Skills & Knowledge: Before training, 33% rated their knowledge and skill level across seven key areas of learning as high. After training, 88% of all participants rated their knowledge and skill level as high. Confidence in Applying Learning: 87% indicated a high-level of confidence in their ability to effectively apply their learning to their jobs. Perceived Value: 97% perceived the training as valuable (i.e., credible, practical, relevant, and essential). Motivation to Apply Learning: 95% rated their motivation to apply their learning as high. Plans for Action: 65% indicated well developed plans for applying their learning in their jobs. (Note: In subsequent studies, a sixth measure, Workplace Readiness, has been added to the Capability Questionnaire.) Capability Index Impact at a Glance Red LOW (1 & 2) Yellow MODERATE (3) Green HIGH (4 & 5) (5-POINT SCALE) SKILL& KNOWLEDGE Before Training 19% 48% 33% After Training 13% 87% CONFIDENCE 13% 87% PERCEIVED VALUE 3 97% MOTIVATION 5 95% PLANS FOR ACTION 35% 65% RISK ALERTS PLANS FOR ACTION APPROXIMATELY 1/3 OF PARTICIPANTS (35%) LACK WELL- DEVELOPED PLANS FOR APPLYING THEIR LEARNING AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 4

5 Effective Practices Using the Effective Practices Audit the following strategies and practices were identified by participants as having contributed to the training event s success: Using face-to-face instruction Group discussions initiated during training Making it a priority for the managers to attend training regardless of project work Using the risk management case studies during training Hosting training at an off-site location Risk Alert for Transfer Approximately one-third of participants reported that they lack well-developed plans for applying their learning. If participants, at the end of their training, lack clear plans for action, the likelihood of their transforming capability into improved performance is at risk. This would suggest an opportunity for course improvement. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 5

6 TRANSFER RESULTS Evaluation Questions: Did the work environment and learning strategy support the transfer of learning to the job? Did training participants apply their learning to their jobs, and did the application of learning impact their work or job performance? To assess Transfer the second link in the Learning Value Chain training participants were asked to complete the Transfer Questionnaire (TQ) three to four months following their training. 72% percent of training participants completed the questionnaire (a good response rate). Questions were also added to the TQ to investigate the extent to which participants completed the "action items" they had planned during training to apply their learning. A high degree of learning transfer occurred for half of those trained The results suggest that for about half the group, the training delivered substantial value at this stage in the Learning Value Chain. They report a high degree of learning transfer and performance improvement an important pre-condition for business improvement. The Transfer Index (at right), summarizes results: Transfer Index Transfer at a Glance Red LOW (1 & 2) Yellow MODERATE (3) Green HIGH (4 & 5) LEARNING APPLICATION 9% 36% 55% (5-POINT SCALE) Learning Application: Across the six key areas of learning, 55% of respondents reported a high degree of learning application; 36% reported applying their learning to a moderate degree; 9% to a low degree or not at all. Performance Improvement: 50% reported a high level of performance improvement on the six key project management areas addressed in training; 37% reported moderate improvement and 13% little or no improvement. PERFORMANCE IMPROVEMENT 13% 37% 50% TRANSFER ALERTS 50% REPORT A LOW OR MODERATE LEVEL OF PERFORMANCE IMPROVEMENT ON THE JOB FOLLOWING TRAINING. BARRIERS 58 % CITE A "LACK OF TIME OR HAVE OTHER HIGHER PRIORITIES" AS THE GREATEST BARRIER TO LEARNING APPLICATION IMPROVEMENT. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 6

7 While 54% of respondents with 16 years or less experience reported a high level of performance improvement; only 35% of those with over 16 years experience reported this level of improvement. Action Plan Completion: 50% of respondents completed the action items developed during training to a great or very great extent. Participants most frequently completed actions related to "Increasing Quality." Only 33% of respondents completed planned actions related to "Decreasing Total Collection Period." 52% completed planned actions related to "Increasing Profitability." Managers with more than 16 years of experience indicated considerably less follow-through on action items compared with managers with less experience. Risk Alert for Business Improvement "Lack of time" a barrier to learning application and improvement Effective Practices Audit As indicated by the red flag in the Risk Alerts box of the Transfer Index, 50% of respondents reported moderate or less improvement in their project management skills a potential risk to business improvement and positive return on investment. Also, reported in the Risk Alert box, the most frequently cited barrier to learning application and performance improvement was: "lack time or have other higher priorities" (cited by 58% of respondents). Using the Effective Practices Audit, the following practices were identified by participants as having helped them to apply their learning and improve their project management practice: Having clear performance expectations Having access on the job to relevant information, reference material, tools and job aids Having acquired sufficient level of knowledge and skills in the initial training AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 7

8 BUSINESS RESULTS TRARA PM 201 training delivers value to the business Evaluation Questions: Did the application of learning or improvements to job performance impact business results? What were the intangible benefits from the training? To evaluate the third link in the Learning Value Chain, participants performance records were reviewed to determine how business performance outcomes (Profitability and Total Collection Period (TCP)) changed after training compared to the previous year, and by how much. In addition, the business impacts were isolated, using estimates from participants and senior management, to determine how much of any improvement should be directly attributed to the training. In the eight months following training, Dillon s business results improved substantially compared with the same period in Participants and senior management independently drew similar conclusions, attributing about one fifth of the overall business improvement to the training. The return on the training investment was calculated to be 28%. In addition, participants cited other key intangible benefits (improved teamwork and improved communications). BUSINESS MEASURES To evaluate training s impact on the target group s performance, individual business performance records were examined for the period ending October 31, This provided a record of the key business metrics over the eight month period following the training. The 2008 performance data were compared to the 2007 data for the same period. The results for each period were annualized extrapolated to reflect business benefits over the full year after training. Profitability: To determine the profit improvement for all Target PMs, each individual s increase (or decrease) in profitability (a percentage value) was multiplied by their revenue numbers. The Total Profit Improvement for the Target Group is the sum of the individual contributions of all participants. Profit Improvement from improved PM profitability: $252, Total Collection Period (TCP): The business contribution from TCP (cost savings) for each project manager was developed by multiplying the total profit improvement for each individual by a standard value determined by the organization (5%). Profit Improvement from TCP: $252, X 5% = $12,616 Total Profit Improvement is the sum of individual profitability and TCP. Total Profit $252, $12, = $264,935 AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 8

9 INTANGIBLE BENEFITS A key component of the Learning Value Chain includes the intangible benefits resulting from the training. Intangibles often yield a richer, nuanced insight into additional benefits of the training which are usually difficult to convert to monetary value. In particular, participants of the PM 201 training identified the following intangibles: Improved Teamwork: More than half of participants (53%) reported that the training improved teamwork to a Great Extent. Improved Communications: Almost two thirds of Project Managers (62%) reported that PM 201 training helped improve communications to a great extent. (Note: These two findings reflect opinions reported by all project managers at all levels of experience.) Training Isolation Questions: What other factors contributed to business results? What proportion of impact can be attributed to training? ISOLATION Another key step when following the Learning Value Chain to ROI is the isolation process which allows us to determine how much of the business impact is directly attributable to the training program. This step acknowledges that other factors may also have contributed to business impact and provides the methodology to quantify, or allocate, training s contribution. This study employed two isolation techniques from the Phillips ROI Methodology : participants estimates and senior managements estimates. Using focus groups and interviews, this approach relies on the assumption that participants are fully capable of developing estimates of training s contribution since they applied the new learning and are in a position to observe its consequences first hand. Training participants attributed 19.6% of the business improvement to the training while senior management attributed 20.9% to the new learning. Based on these estimates of the two groups, the profit contribution from the training is as follows: Participants: $264, X 19.6% = $51,927 Senior Management: $264, X 20.9% = $55,371 In this case the participants estimate of profit contribution ($51,927) is selected since it is the lowest, and is thus the most conservative option available. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 9

10 ROI Evaluation Questions: Did the net business benefits of training exceed its total costs (Benefit/ Cost Ratio)? What was the return on investment? COST The cost of the training, a key component of the ROI, is the sum of all expenses related to the training process including program development, travel, accommodation, food, and the fully-loaded salaries of the facilitators and training participants for the time in training. Training Cost: $40,653 (Based on 38 participants at $1, per participant.) BENEFIT COST RATIO (BCR) A key metric used in the IIP Methodology, the Benefit-Cost Ratio permits a ready comparison of the programs benefits with the program costs. Benefit Cost Ratio (BCR) = (Total Benefits) (Total Costs) $51,927 BCR for PM 201 training: = 1.28:1 = 1.28 $40,653 A BCR of 1.28 means that for every dollar spent on PM 201 training $1.28 was returned to the organization in business revenue. RETURN ON INVESTMENT (ROI) Return on investment (ROI) is calculated as follows: (Total Benefits Total Costs) ROI = X 100% (Total Costs) ($51,927 - $40,653) ROI for PM 201 training = X 100% = 27.7% ($40,653) AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 10

11 CONSIDERATIONS & ISSUES 1. An ROI of 28% means the original investment in training the target group was returned and a 28% profit was earned in addition. This means every dollar spent on PM 201 Training is returned to the organization plus a profit of $ The guidance of the Phillips ROI Methodology TM recommends that only the first year of benefits be recognized in the calculations (annualized). This approach is conservative since it discounts any impact that may be realized in subsequent years. In fact, however, it is very likely that the participants knowledge and skills will continue to deliver improved business performance and continue delivering returns on the original investment for some time in the future. 3. The small size of the target group (38 participants) compared to the full staff means that the results cannot be considered statistically significant (as most often is the case with ROI methodology). Nevertheless, this tried and true methodology trades off scientific precision for with an evidence-based process that is methodologically conservative and requires a minimum of valuable resources (time and money). It is a reasonable assumption that most of Dillon s Project Managers who took the same training would realize similar performance improvements as did the target group (approximately 20% of the total number of Project Managers). 4. This study observes conservative principles in decisions involving estimation. When more than one option is available relating to benefits, the lowest estimate is always taken. Conversely, when several options or estimates are available pertaining to training costs, only the highest value is selected. (For example, it could be argued that senior management s estimates of the percent of training contribution to improvement would be more informed and accurate. Nevertheless, the study conservatively chose the lower option, the training participants estimates.) 5. The study accounted for other factors that might also have contributed to the business improvement observed subsequent to training. Participants were asked to suggest other factors that might also have contributed to the improvement. Once all other potential factors were identified, participants were then asked to assign percentages to each factor according to how much they believed each factor contributed to the training. Thus the estimate of training s contribution does not take place in a vacuum but within the context of careful and balanced consideration of all other key factors that might also have contributed to the observed success. Risk Alert for ROI PMs with most experience received less value from the training Analysis of focus group data suggests that the more experienced project managers perceived less value in the PM201 training than did those with less tenure. Project managers with greater than 16 years experience attributed a lower estimate of the percentage impact of training (15%) which, if accepted, results in a lower estimated ROI (0%) compared with all PMs as a whole. Interestingly, those with 16 or less experience had a considerably higher rating of the training s value. This group attributed 25% of the performance improvement to the training, estimating a profit contribution of $64,114 resulting in an ROI of 62%. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 11

12 Viewed strictly from the perspective of improving ROI, for those with more than 16 years of experience, attending PM 210 training adds additional costs with little likelihood of a corresponding improvement in business performance. LESSONS LEARNED You can t teach an old dog new tricks. - Senior Project Manager ISSUE Closer analysis of participant estimates of business improvement and subsequent ROI appears to confirm the observations made in Transfer phase of the evaluation that PM 201 training was not targeted at the learning needs of those with more than 16 years of experience. RECOMMENDATION 1: When seeking to enhance training ROI, it is recommended that future plans for delivering PM 201 and other training initiatives match the learning content with the learning audience and consider the varied learning needs of prospective participants. (Note: In this case, it is possible that the presence of senior project management staff in attendance at the PM 201 training may have delivered significant intangible benefits more than offsetting their training costs. They may well have delivered benefits to the less experienced participants by enriching discussion with deep reservoirs of experience and expertise. ISSUE Some participants suggested that some of PM 201 content might be too abstract and would benefit by being anchored in real world situations. I would put less emphasis on the theoretical parts of the training and more on practical studies. Should be more hands on with the tools we have (CFMS). I think there is an opportunity to provide staff with more concrete strategies to maximize profits on a job. Although there is good discussion of Dillon s cost structure and the way profit is calculated, there is little discussion on specific and small items which can maximize or even erode profit. RECOMMENDATION 2: Participants suggestions pointed to the opportunity to enhance the key metrics (TCP and Productivity) by focusing more on the tools and situations that reflect the Dillon experience. More of the learning could draw on the organization s rich case history using actual Dillon projects as examples. ISSUE Transfer Risk Alert flagged the concern that some 50% of participants reported a low or moderate level of performance improvement on the job after training. Since Transfer to the job is the weakest link in the Learning Value Chain TM, addressing this risk is possibly the surest opportunity to enhance the PM 201 s prospects for success and deliver improved bottom line outcomes including ROI. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 12

13 Two other Alerts warned of the risk for sub-optimal business outcomes from the training investment: The Capability Index signaled the looming problem of transferring the training to the job reporting that Approximately 1/3 of participants (35%) lack well-developed plans for applying their learning. The Barriers component of the Transfer Index also singles out another very common and notorious factor militating against success: 58% cite a lack of time or have other high priorities as the greatest barrier to learning application improvement. Emphasize Action Planning Provide Time and Opportunity RECOMMENDATION 3: PROMOTE ACTION PLANNING It is suggested that Dillon might consider placing greater emphasis on the Action Planning process. Experience has shown that if individual learners develop strategies and intentionality to apply the new knowledge and skills to their own projects, the learning is more lasting and effective. Because of the intense competition for project managers attention once they return to the the job, action planning has the greatest prospects for success if it is paired with strategies to promote and ensure participants accountability in completing action items identified. RECOMMENDATION 4: PROVIDE TIME FOR LEARNING TRANSFER Similarly, if improvement of business performance measures and ROI is the goal of training, participants must be allowed time and opportunity to assimilate the new learning into their practice routines. Due consideration should be accorded to the post-training work environment. On the other hand, if an individual s work agenda is too pressing, too time-constrained, and too critical to the business, it is worth reflecting on the wisdom of investing in training in the first place. Learning theory abounds with evidence that to effectively transfer new learning to the job, participants must have robust opportunities to apply the new skills. The authors wish to thank Lucy Jakupi, Training Manager and Associate at Dillon Consulting, for her assistance in facilitating and coordinating evaluation activities in her firm and making this training evaluation possible. AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 13

14 AUTHORS: Lynette Gillis & Allan Bailey, Centre for Learning Impa 2009 CSTD Page 14