Evaluating effects from workplace training and employee development (TED)

Transcription

1 Evaluating effects from workplace training and employee development (TED) Annual HEAD Conference June 11th, 2007 PhD Student Anders Dysvik Department of Leadership and Organizational Management Norwegian School of Management BI

2 Definition and current interest (Tannenbaum, 2002; Kraiger & Ford, 2007) Training and employee development (TED) is defined as a systematic approach to learning and development to improve individual, team, and organizational effectiveness (Kraiger & Ford, 2007, p. 281). Workplace settings are becoming increasingly complex Pressures from both public authorities and customers in terms of improved solutions and products Organizations must be attractive for new employees and simultaneously prevent existing employees from leaving the organization An increasing trend that TED supplements formal education in workplace settings at the employer s expense TED advocates are facing increasing demands in demonstrating return on investments from such activities 59 per cent of employees between ages participated in TED during their last 12 months of employment (Nyen & Skule, 2005) Annual TED spendings have been estimated to 17,7 Billion NOK, or 2,8 Billion USD (Nordhaug, Gooderham & Døving, 2004)

3 Scope Lay theories suggest a causal relation between investments in TED and organizational performance at macro level (Sels, 2002) Norwegian companies are at the lower end of the scale in systematically assessing the return of investment relating to TED (Nordhaug & Nordhaug, 2004; Lai, 2004). Majority of TED studies limited to needs assessment, design, implementation, evaluation (Salas & Cannon-Bowers, 2001; Wright & Boswell, 2002). TED at this level is one of the most widespread phenomena studied within the HR domain (Whitfield, 2000) Cross-sectional designs insufficient in arguing for causal relationships (Wright, Gardner, Moynihan & Allen, 2005). Cross-sectional designs establishing positive relationships often lack control of performance prior to participation. When included, past performance is highly correlated with performance following TED (Arthur jr., Bennett jr., Edens & Bell, 2003; Morin & Renaud, 2004; Maurer, Weiss & Barbeite, 2003)

4 Causal assumptions It has been argued that TED is of little or no value for organizations, in that more than 70 per cent of workplace learning are informal events in the daily setting (Pfeffer & Sutton, 2000) Vast parts of acquired knowledge and skills may fade into oblivion (Arthur jr., Stanush, Bennett & McNelly, 1998) Contrary, recent meta-analyses suggest a standardized rho of 0.60 between TED and performance (Arthur et al., 2003) TED should be assessed based on employee perceptions and in relation to the bundle of HR activities (Tannenbaum, 2002; Lee & Bruvold, 2003; Wright, 2002) Managerial or HR assumptions of TED are not always in line with employee perceptions of the same practices (Edgar & Geare, 2005) TED should be assessed in relation to the mechanisms it operates, as a number of factors seems to influence participation, acquirement, and transfer of TED content (Kuvaas, 2007; Colquitt, LePine & Noe, 2000) Evidence-based TED as opposed to intentions behind the efforts, hypes or trends which assumedly justify such investments (Pfeffer & Sutton, 2006)

5 (Kraiger, 2003, p. 173)

6 Evaluating effects from TED Both researchers and practitioners agree that assessing TED effectiveness is important There are clear inconsistencies regarding how assessments should be conduced (Alliger, Tannenbaum, Bennett jr., Traver & Shotland 1997). Established evaluation practices are not always based on evidence-based criteria (Kraiger, 2002) The most wide-spread practice seems to be the one which hold the least degree of evidence-based information regarding training effectiveness (Noe, 2002)

7 Are TED effective? For the last forty years, the dominant tool has been Kirkpatricks hierarchical model (1959a; 1959b; 1960a; 1960b) Practical guidelines based primarily on Kirkpatrick s own experiences in a time when few other models were available, Although revised and presented in later writings, the model has not changed significantly. Given the widespread acceptance of the Kirkpatrick hierarchy, any reasoned discussion of training evaluation should embark here (Kraiger, p. 333). Higher levels should not be evaluated if satisfactory results at lower levels have not been attained Direct relationships between the levels of the model Changes at the higher levels are more valuable than at lower levels Learning positioned in an behavioralistic tradition

8 (Fischer, Schoenfeld & Shaw, 2006, p. 405)

9 Are TED effective II? Scarce support for assuming that lower levels are of less concern than higher levels The field of learning psychology has evolved since Lack of conceptual clarity within and between levels Low or absent direct relations between levels 1 and 2 In a recent study of evaluation practices among 367 U.S. companies, 78% used level 1 measures, 32% level 2 measures, 9 % level 3 measures, and 7% level 4 measures (Van Buren & Erskine, 2002). The widespread use of reaction measures may be explained in that they are easy to acquire. When regarded in isolation, such reaction measures fail to explain the alleged relation between employee reactions and consequent performance at individual, group, and organizational level (Haccoun & Saks, 1998).

10 Are TED effective? III Affective reactions such as I liked the course and consequent performance.02 (Alliger et al. 1997),.10 reactions and knowledge acquisition/.09 reactions and skill acquirement (Colquitt et al., 2000). Recently replicated in a meta-analysis of 162 TED studies (Arthur jr. et al., 2003) Most reactions are positive, whereas, negative reactions demonstrate a positive, albeit weak relationship with performance (Schmidt & Björk, 1992) How should effects be assessed, do you wait for positive reactions to occur at level 1 before proceeding to level 2? A strong negative relationship between levels 2 and 3, which call for a more integrative perspective (Arthur et al., 2003; Colquitt et al., 2000) Assuming impact at level 4 becomes problematic as content increases in complexity, in addition to: Competing hypotheses? Controlling for past performance? In the most systematic efforts as of yet, no conclusions could be drawn between the different levels and impact at macro level (Alliger et al., 1997)

11 Moving on: Decision-based evaluation (Kraiger, 2002; Twitchell, Holton & Trott, 2001) What is the purpose for evaluation? Present evidence for decision-making Feedback to designers and trainers Internal and external marketing What are the obstacles to evaluation? Not required by the organization Additional costs associated with evaluation Lack of time Lack of training in how to evaluate Training function may have everything to lose and nothing to gain from the data

12 Moving ahead: Decision-based evaluering (Kraiger, 2002) (Kraiger, 2002, p. 343)

13 Moving ahead Decision-based evaluation II (Kraiger, 2002) Training content and design Formative purposes Reaction measures I liked the course.02, while I found the course content relevant.26 (Alliger et al., 1997) Novice participants may limit the latter (for instance student s evaluation of teaching) Training environment, transfer expectation, logistics and administration, overall evaluation, suggestions for improvement

14 Moving ahead Decision-based evaluation III (Kraiger, 2002; Kraiger, Ford & Salas, 1993) Changes in the learner Cognitive outcomes Affective outcomes Behavioral outcomes Changes not necessarily the same as improved job performance, as this is inflicted by a number of influences Level of retention over time Organizational payoffs Evidence that trainees are applying learned behaviors or skills back at work These behaviors result in more effective performance TED has bottom-line impact on unit or organizational effectiveness

15 Decisions regarding design Agreement between trainers and TED advocates regarding the level of evidence before initiation Arguing for effects of TED beg the presence of experimental designs, often difficult to conduct in workplace settings (Morin & Renaud, 2004) For those without a scolarly approach, insurmountable or not a problem at all For those with a scolarly approach, a challenging endevaour Often conducted as quazi-experimental designs Worst case scenario: Only immediate post-tests, and still arguing for effects from TED Examples of alternative designs Internal Reference Strategy (Haccoun & Saks, 1998) Level of design evidence should be in line with the strenght of claims made by TED advocates Less formal designs may still serve a purpose, and is far better than direct linkage of reactions and effectiveness, or failing to evaluate all together

16 References Alliger, G. M., Tannenbaum, S. I., Bennett jr., W., Traver, H., & Shotland, A. (1997). A Meta-analysis of the relations among training criteria. Personnel Psychology, 50, Arthur jr., W., Bennett, W., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative review and analysis. Human Performance, 11(1), Arthur jr., W., Bennett jr., W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of Training in Organizations: A Meta-Analysis of Design and Evaluation Features. Journal of Applied Psychology, 88(2), Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85(5), Edgar, F., & Geare, A. (2005). HRM practice and employee attitudes: different measures - different results. Personnel Review, 34(5), Fischer, C. D., Schoenfeldt, L. F., & Shaw, J. B. (2006). Human Resources Management (6th ed.). London: Houghton Mifflin Company. Haccoun, R. R., & Saks, A. M. (1998). Training in the 21st century: Some lessons from the last one. Canadian Psychology-Psychologie Canadienne, 39(1-2), Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of ASTD, 13(11), 3-9. Kirkpatrick, D. L. (1959). Techniques for evaluating training programs: Part 2 - Learning. Journal of ASTD, 13(12),

17 References II Kirkpatrick, D. L. (1960). Techniques for evaluating training programs: Part 3 - Behavior. Journal of ASTD, 14(1), Kirkpatrick, D. L. (1960). Techniques for evaluating training programs: Part 4 - Results. Journal of ASTD, 14(2), Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of Cognitive, Skill-Based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation. Journal of Applied Psychology, 78(2), Kraiger, K. (2002). Decision-Based Evaluation. In K. Kraiger (Ed.), Creating, implementing, and maintaining effective training and development: State-of-the-art lessons for practice (pp ). San Francisco, CA: Jossey-Bass. Kraiger, K. (2003). Perspectives on Training and Development. In W. C. Borman, D. R. Ilgen & R. J. Klimoski (Eds.), Handbook of Psychology (Vol Industrial and Organizational Psychology, pp ). Hoboken, NJ: Wiley&Sons. Kraiger, K., & Ford, J. K. (2007). The expanding Role of Workplace Training: Themes and Trends Influencing Training Research and Practice. In L. L. Koppes (Ed.), Historical Perspectives in Industrial and Organizational Psychology (pp ). Mahwah, NJ.: Lawrence Erlbaum Associates. Kuvaas, B. (In press). An exploration of how the employee-organization relationship affects the linkage between perception of developmental human resource practices and employee outcomes. Journal of Management Studies. Lai, L. (2004). Kompetanse som begrep og fenomen. In L. Lai (Ed.), Strategisk Kompetansestyring (2nd ed., pp ). Bergen: Fagbokforlaget.

18 References III Lee, C. H., & Bruvold, N. T. (2003). Creating value for employees: investment in employee development. International Journal of Human Resource Management, 14(6), Maurer, T. J., Weiss, E. M., & Barbeite, F. G. (2003). A model of involvement in work-related learning and development activity: The effects of individual, situational, motivational, and age variables. Journal of Applied Psychology, 88(4), Morin, L., & Renaud, S. (2004). Participation in corporate university training: Its effect on individual job performance. Canadian Journal of Administrative Sciences-Revue Canadienne Des Sciences De L Administration, 21(4), Noe, R. A., & Colquitt, J. A. (2002). Planning for training impact: Principles of training effectiveness. In K. Kraiger (Ed.), Creating, implementing, and maintaining effective training and development: State-of-the-art lessons for practice (pp ). San Francisco, CA: Jossey-Bass. Nordhaug, O., Gooderham, P. N., & Døving, E. (2004). Kompetanseutviklingens jernlov. In O. Nordhaug (Ed.), Strategisk kompetanseutvikling-teori og praksis (pp ). Oslo: Universitetsforlaget. Nordhaug, O., & Nordhaug, I. W. (2004). Kompetansebehov i næringslivet. In O. Nordhaug (Ed.), Strategisk kompetanseledelse-teori og praksis (pp ). Oslo: Universitetsforlaget. Nyen, T., & Skule, S. (2005). Livslang læring i norsk arbeidsliv. In M. Raabe, O. Raaum, P. O. Aamodt, N. M. Stølen & A. M. Rustad Holseter (Eds.), Utdanning deltakelse og kompetanse. Oslo: Statistisk Sentralbyrå. Pfeffer, J., & Sutton, R. I. (2000). The Knowing-doing gap: How smart companies turn knowledge into action. Boston. MA: Harvard Business School Press.

19 References IV Pfeffer, J., & Sutton, R. I. (2006). Hard Facts, Dangerous Half-Truths, and Absolute Nonsense. Profiting from Evidence-Based Management. Boston, MA: Harvard Business School Press. Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52, Schmidt, R. A., & Bjork, R. A. (1992). New Conceptualizations of Practice - Common Principles in 3 Paradigms Suggest New Concepts for Training. Psychological Science, 3(4), Sels, L. (2002). 'More is not necessarily better': the relationship between the quantity and quality of training efforts. International Journal of Human Resource Management, 13(8), Tannenbaum, S. I. (2002). A Strategic View of Organizational Training and Learning. In K. Kraiger (Ed.), Creating, implementing, and maintaining effective training and development: State-of-the-art lessons for practice. (pp ). San Francisco, CA: Jossey-Bass. Twitchell, S., Holton, E. F., & Trott, J. R., Jr. (2001). Technical training evaluation practices in the United States. Performance Improvement Quarterly, 13(3), Van Buren, M. E., & Erskine, W. (2002). The 2002 ASTD state of the industry report. Alexandra, VA: American Society of Training and Development. Whitfield, K. (2000). High-performance workplaces, training, and the distribution of skills. Industrial Relations, 39(1), Wright, P. M., & Boswell, W. R. (2002). Desegregating HRM: A review and synthesis of micro and macro human resource management research. Journal of Management, 28(3), Wright, P. M., Gardner, T. M., Moynihan, L. M., & Allen, M. R. (2005). The relationship between HR practices and firm performance: Examining causal order. Personnel Psychology, 58(2),