UT Center fr Industrial Services Can We Really Measure Training ROI? (D We Really Want T?) Bill Stetar University f Tennessee 2003 DOE Annual Training Sympsium "Transfrmatin fr Success"
UT Center fr Industrial Services Purpse f this Sessin... 1. Examine advancements in mdels that apprach training ROI systematically and rigrusly in rder t gain a truer measurement f training impact. 2. Prvide ways fr training prviders t cnfidently prvide data that supprts justificatin f training frm a business perspective. 3. Describe hw training prgram design shuld start with business results and nt learning bjectives. 4. Distinguish between predictive and retrspective measurement. Stetar - DOE Training Sympsium - Aug 2003 2
UT Center fr Industrial Services Level 1 Reactin 2 Learning 3 Behavir 4 Results The Kirkpatrick Mdel Type f Measure Satisfactin Learning, Attitudes Behaviral change Results Fcus Learner satisfactin with the prgram Principles, facts, techniques Changes in jb behavir Tangible results cst, quality, quantity Stetar - DOE Training Sympsium - Aug 2003 3
UT Center fr Industrial Services Origin f "Training ROI" Key prpnent Jack Phillips Expressed as a rati / percentage % ROI = (Net Benefits / Csts) x 100 Level 5 measurement Effrt t speak the vernacular Gain acceptance & credibility Stetar - DOE Training Sympsium - Aug 2003 4
UT Center fr Industrial Services Why ROI? Abut $5.6 billin t $16.8 billin is wasted annually n ineffective training prgrams American industry is spending billins and billins n training prgrams and ding n evaluatin f their effectiveness. Yu have t measure it. - Cary Cherniss, Rutgers University 10/07/1998 Stetar - DOE Training Sympsium - Aug 2003 5
UT Center fr Industrial Services Why ROI? A study f 15 cuntries in the Organizatin fr Ecnmic Cperatin and Develpment fund that the majrity f enterprises believe emplyee training is respnsible fr "prductivity imprvements, greater wrkfrce flexibility, savings n material and capital csts, imprved quality f the final prduct r service, and a mre mtivated wrkfrce." Hwever, many cmpanies have nt measured the benefits and related them t the cst f training in a way that reveals the rate f return n a firm's investment. Apparently there is n ther wrkplace issue n which s much mney is spent with as little accuntability as training. Myths & Realities N. 16: Return n Investment in Training, Bettina L. Brwn, Center n Educatin & Training fr Emplyment, Ohi State University, ERIC/ACVE, 2001 Stetar - DOE Training Sympsium - Aug 2003 6
UT Center fr Industrial Services Critics f Level 5 Say... N mre than a restatement f Level 4 results Executive management has nt bught int the cncept ROI des nt measure wrth r "sft" benefits, it is nly ne tl that can measure results (Ernst & Yung) Accunting wrld des nt view training as a capital investment Stetar - DOE Training Sympsium - Aug 2003 7
UT Center fr Industrial Services ROI vs Cst-Benefit Analysis Surce: Measuring Return n Investment (ROI), David P. Wegenast, D.S.W., Prceedings f the Furth Annual Natinal Human Services Training Evaluatin Sympsium, May 23-25, 2001, UC Berkeley "Ding training ROI analysis is a fancy way f ding cst-benefit analysis." - Olga Bulatva, Direct f Ernst & Yung Prfessinal Educatin Center in the CIS Cuntries Stetar - DOE Training Sympsium - Aug 2003 8
UT Center fr Industrial Services N tw mdels are alike... Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 9
UT Center fr Industrial Services Mst Have These In Cmmn Direct & Indirect Csts Attributable t the Prject r Prgram Benefits Must be Cnverted t Equivalent Mnetary Units Measurement is Abslutely Critical Baseline & Pst- Mrtem ROI r CBA must be reserved t missin-critical prgrams Assumptins, assumptins, & assumptins Stetar - DOE Training Sympsium - Aug 2003 10
UT Center fr Industrial Services Merck Mdel Develped early-1990's in respnse t CEO's inquiry: "What are We Getting fr Our Mney?" Situatin at Merck: 1986 $25 millin spent n training 1991 - $70 millin spent n training Stetar - DOE Training Sympsium - Aug 2003 11
UT Center fr Industrial Services Merck Mdel (cnt'd) Cnstructed its wn versin f the "Utility Mdel" Value f an activity is derived based n statistical infrmatin and a clear understanding f hw the business perates. Stetar - DOE Training Sympsium - Aug 2003 12
UT Center fr Industrial Services Key Terms: Validity & Reliability Valid: measures what it is suppsed t measure Reliable: cnsistent and reprducible Stetar - DOE Training Sympsium - Aug 2003 13
UT Center fr Industrial Services What are the trends? The basic Kirkpatrick, Cst-Benefit mdel appears t have widespread usage r applicatin Predictive measurement is as crucial as retrspective measurement Causal analysis methds being used t link results t actin Training design predicated by quantifying desired business results Stetar - DOE Training Sympsium - Aug 2003 14
UT Center fr Industrial Services Trends (cnt'd) Shift in emphasis: Frm "training" t "learning" Key prpnents: Mtrla, IBM & Xerx (t name a few) http://www.ibmweblectureservices.ihst.cm/servlet/gate/cmpnent?actin Stetar - DOE Training Sympsium - Aug 2003 15
UT Center fr Industrial Services Where's the Lw Hanging Fruit? Due Diligence training yu are required t d either by law r mandate Business imprvement training prvided t increase skill capacity, cunteract perfrmance gaps, supprt business gals Stetar - DOE Training Sympsium - Aug 2003 16
UT Center fr Industrial Services Fr "Due Diligence" Activities, typically, the measurement is internally fcused Fr example. Cst per student Travel cst reductins Material csts Supprt csts Emphasis is n "Cst-Avidance" ROI Stetar - DOE Training Sympsium - Aug 2003 17
UT Center fr Industrial Services "Business Imprvement" Activities are Ideal Candidates fr ROI Analysis ROI Perspective Specifies desired utcmes befrehand Captures baseline data Establishes measurement criteria what & when Identifies ptential variables Stetar - DOE Training Sympsium - Aug 2003 18
UT Center fr Industrial Services Benefits Target Key Areas Prductivity & Efficiency Sales & Prfitability Quality f Prducts & Services Custmer Service & Satsifactin Health & Safety Organizatinal Learning & Develpment Organizatinal Climate, Culture & Practices Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 19
UT Center fr Industrial Services Prductivity & Efficiency prductin csts per unit prductivity targets met/exceeded prductin/cmpletin time per unit (e.g. frms, lans, clients, prjects) utput (per wrked hur, per shift, per machine, r per annum) vertime (quantity, cst) imprved innvatin in prducts/services inductin time fr new emplyees prductivity f new emplyees equipment/facility/asset utilisatin (e.g. dwn time due t machine stppages, shift changever time) equipment maintenance (csts r repair time), r replacement csts respnse time (e.g. t service calls r rders) capacity f staff t slve rutine and nn-rutine prblems (e.g. supervisin time required) staffing requirements and wrkfrce flexibility (e.g. dependence n casual/cntract labur) Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 20
UT Center fr Industrial Services Sales & Prfitability verhead csts perating csts perating csts as a percentage f ttal csts/revenue value f cntracts wn, lans prcessed revenue/incme/sales (mnthly, annually, per emplyee, per team, per branch r stre) market share (number f custmers, dllars spent, unit vlume sld) sales t new custmers grup perating prfit prfit per emplyee stck market perfrmance (i.e. sharehlder return) Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 21
UT Center fr Industrial Services Quality f Prducts & Services n time prvisin f prducts/services wastage, reject, errr r rewrk rates cnfrmance recrd with quality specificatins (e.g. batch yields, thrughput f invices) achievement/maintenance f quality rating cmpliance with quality, legal and/r ethical requirements achievement f quality award Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 22
UT Center fr Industrial Services Custmer Satisfactin & Services custmer satisfactin levels (with timeliness, availability, quality and price f gds and services) custmer relatinships and experiences repeat business (custmer retentin r lyalty) new business resulting frm client referrals mre/new custmers r markets (e.g. cntracts wn, lans prcessed, funding awarded) lst business number f cmplaints Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 23
UT Center fr Industrial Services Health & Safety accidents r injuries (number, time lst, cmpensatin csts, premium cst/rating) safety critical incidents (number, cst) cmpliance with safety and health requirements vilatin f safety rules imprved respnse t crises Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 24
UT Center fr Industrial Services Organizatin Learning/Develpment perfrmance appraisal ratings achievement f rganisatinal cmpetency prfile requirements (e.g. t meet accreditatin r licensing requirements, new perating envirnments r facilitate rganisatinal expansin) number/percentage f emplyees with natinally recgnised qualificatins internal prmtins resulting frm emplyee cmpetence and perfrmance training awards received emplyee perceptins f training and develpment pprtunities alignment with human resurces, business and strategic planning Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 25
UT Center fr Industrial Services Organizatin Culture/Climate emplyee retentin/turnver/recruitment (e.g. numbers, csts) absenteeism disputes/grievances (number, cst r time lst) number f emplyee suggestins (submitted r implemented) emplyee satisfactin and mtivatin interpersnal relatinships and cmmitment t team gals participatin in teams and cmmittees team perfrmance internal cmmunicatin and infrmatin systems implementatin f new wrk practices standardisatin f wrk practices implementatin/maintenance f a service culture cntributin t re-engineering and refcussing f enterprise Surce: Shwing That Enterprise Training Pays Off, Janelle My, NCVER Stetar - DOE Training Sympsium - Aug 2003 26
UT Center fr Industrial Services ROI Starts with FEA First & fremst: determine if there is a prblem r a perfrmance gap wrth ding smething abut Quantify the cst f the prblem Specify a slutin that csts less than the prblem Specify the utcmes f any training interventin t be applied Set up yur measurement methdlgy validity & reliability It's like risk analysis Stetar - DOE Training Sympsium - Aug 2003 27
UT Center fr Industrial Services $Benefit Prgram Benefit 0 $Cst Prgram Cst t 1 t 2 Time Measuring ROI at time t 1 will nt be able t justify HRD investment Surce: Greg Wang, PhD, DPT Cnsulting Grup, Inc. 2000 Stetar - DOE Training Sympsium - Aug 2003 28
UT Center fr Industrial Services Pay me nw r pay me later Evaluate & Imprve Implementatin Develpment Design Analysis Stetar - DOE Training Sympsium - Aug 2003 29
UT Center fr Industrial Services Target Audience Des Matter! A FRESH LOOK AT RETURN ON INVESTMENT A White Paper by Jay Crss, 2000, SmartFrce Stetar - DOE Training Sympsium - Aug 2003 30
UT Center fr Industrial Services Thank yu Bill Stetar University f Tennessee bstetar@tennessee.edu 615-532-4910 Stetar - DOE Training Sympsium - Aug 2003 31