Using Data to Help improve a Return on Investment

Transcription

1 Using Data to Help improve a Return on Investment Amar P. Patel, MS, NREMT-P, CFC Director Center for Innovative Learning WakeMed Health & Hospitals Raleigh, North Carolina

2 Overview ROI ROE Educa/onal Principles & Kirkpatrick Data in simula/on courses Link simula/on to ROI Calcula/ons and examples Where can ROI be calculate from? ROE or ROI?

3 Objec/ves Define Return of Investment (ROI). Describe how to calculate Return of Investment. Discuss how to link simula/on and ROI. Demonstrate an ROI calcula/on u/lizing informa/on provided through audience feedback. Determine 5 key areas ROI can be calculated from.

4 Questions? How do we utilize existing technology to integrate a multi-disciplinary group? How do we utilize simulation to teach multiple disciplines? What are these opportunities?

5 A simple understanding of history

6

7

8

9

10

11

12

13

14

15 Average Age 38 Entertainment Software Association. (2010). Industry facts. Retrieved from

16 Growing Issues Medical procedures are becoming more numerous and more complex medical knowledge has hypertrophied. (Cooke, et al, 2006) Clinical rota/ons are becoming more difficult to schedule. Clinical seyng is not the place to prac/ce skills. Cooke, M., Irby, D., Sullivan, W., & Ludmerer, K. (2006). American medical educa/on 100 years a]er the flexner report. The New England Journal of Medicine, 355(13),

17 What does simula/on offer? Repe//on which leads to proficiency Understanding of the cogni/ve and the prac/cal Training opportuni/es Established objec/ves Clear expecta/ons Quality & Quan/ty Immediate applica/on

18 Funding How do you secure funding? What does that mean? For what period of /me? Can a simula/on program be self- sufficient? What do investors want? Repor/ng on funding?

19 Consider these costs

20 Developing a sim center Are you looking to develop a new building or use an exis/ng building? A cost is based on program objec/ves? short & long term goals? your target audience? Why does it majer?

21 Investments Program investment = $1 million Student count = 1,000 people/year Cost per student = $1, What about intangibles? Long term expenses? Supplies? Gases? What does that cost? What are the true financial needs?

22 Intangibles Community Outreach Dona/ons High visibility show you mean business! Does there need to be a business model? What is free and what costs money?

23 Is simula)on profitable?

24 Is it?

25 Mo/ve 1: Lower Cost } Powerful Mo>vators Mo/ve 2: Bejer Access to Symptoms/Cases Mo/ve 3: Reduced Training Time Mo/ve 4: Reduced Errors Smith, R. (2009). Embracing game technology for medical educa/on. Retrieved from hjp://

26 Business Plan

27 Defined as: A wrijen document which describes the business, its objec/ves, its strategies, the market in which it operates and its financial forecasts. (Markow, 2011) Markow, M. (2011). Business terms glossary. BusinessWings. Retrieved from hjp:// terms- glossary

28 Consists of: Business Plan Execu/ve Summary* Market Analysis Company Descrip/on Organiza/on & Management Marke/ng & Sales Management Service or Product Line Funding Request Financials Appendix SBA.gov. (2011). Essen/al elements of a good business plan. Retrieved from hjp:// structure /star/ng- managing- business/star/ng- business/wri/ng- business- plan/essen/al- elements- good- busines

29 Remember this: hospitals typically measure ROI from a business perspec/ve cost, revenues or opera/ng efficiencies but many benefits of clinical applica/ons fall into quality and safety realms that do not easily translate into dollars. (Page, 2010) Page, D. (2010). IT s return on investments is tricky to pin down. Hospital & Health Networks. Retrieved from hjp:// /ar/cledisplay.jsp?dcrpath=hhnmag/ar/cle/data/06jun2010/1006hhn_fea_mostwired&domain=hhnmag

30 A business plan will only make you stronger

31 The reality is: "We should measure clinical ROI in terms of measurable impact on pa/ent care. (Page, 2010) Page, D. (2010). IT s return on investments is tricky to pin down. Hospital & Health Networks. Retrieved from hjp:// /ar/cledisplay.jsp?dcrpath=hhnmag/ar/cle/data/06jun2010/1006hhn_fea_mostwired&domain=hhnmag

32 A Return on Investment

33 Tradi/onal ROI is: performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. ( Investopedia, n.d.) if an investment does not have a posi/ve ROI, or if there are other opportuni/es with a higher ROI, then the investment should be not be undertaken. ( Investopedia, n.d.) Investopedia. (n.d.). Return on investment - ROI. Retrieved from hjp://

34 Solu/on Matrix. (2011). Return on investment. Retrieved from hjp:// on- investment.html

35 Tradi/onal ROI ROI evaluates an investment s poten/al by comparing the magnitude and /ming of expected gains to the investments costs. ( Best, 2002) ROI = gains investments costs Investments costs $700,000 (saved) $500,000 sim center $500,000 = 40% Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp:// /thevalueof_it_investments.pdf

36 Healthcare ROI is: the amount of improvement in care brought about by a certain investment. ROI can also refer to the theory that if you invest in health care quality now, then the quality of care for pa/ents will improve in the future. ( Robert Wood, 2011) Robert Wood Johnson Founda/on. (2011). Glossary of healthcare terms. Quality/Equality. Retrieved from hjp://

37 Healthcare ROI 1. Examines impact to system processes. 2. Determines a cost to each process related change and if the cost was favorable. 3. That cost is consider the return on investment. ((# Process errors found x Es/mated cost per error)- SIM COSTS) x (% $ Paid Out for errors) SIM COSTS

38 Healthcare ROI ROI Total: 5.93% Number of Process Errors Found: 14 Es/mated Cost per Error: $35, } % Hospital $ Paid Out: 0.25% SIM Costs: $19, $490,000

39 Healthcare ROI Issues Who is actually providing the training that is resul/ng in the ROI being created? How are you determining process issues? Are there clear program objec/ves? How are you being funded? Who benefits from the ROI? Who are your stakeholders?

40 Return on Investment Defensive isola/on approach that separates func/ons. Defines training as an end in itself. Value defined by predetermined formula. Focus on single metric of numeric proof. Complex, rigid, and expensive. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

41 Let s try a few calcula)ons

42 To create ROI, we must develop a value measuring methodology. ( Best, 2002) Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp:// /thevalueof_it_investments.pdf

43 Value Measuring Methodology 1. Develop a decision framework Customers Social Government Opera/onal/Founda/onal Strategic/Poli/cal Government Financial 2. Perform an alterna/ve analysis 3. Gather all the informa/on 4. Communicate and document Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp:// /thevalueof_it_investments.pdf

44 What does all this mean? There has been this ongoing debate in medical simula/on about the ability to prove ones worth - then been able to u/lize that proof to secure addi/onal funding from donors. Does simula/on make an impact in educa/on? Does it make an impact in healthcare, in pa/ent safety, in changes in human behavior or system processes? There are numerous issues that result from trying to jus/fy that simula/on was the sole reason a system benefited. Was it the single cause or was there other educa/onal programs that helped shaped changes? What happens if the data shows decline? Was simula/on to blame?

45 The mo/va/on The want and desire to include simula/on. It s the in thing to do The data shows improvement in cogni/ve learning. The data shows improvement in prac/cal applica/on and reten/on.

46

47 But

48 You have to think through it!

49 Business side Who are your key stakeholders? Internal External Have you considered developing a business plan? Do you have opera/onal & strategic goals? Who is managing your program data & finances?

50 When can I see a return?

51 Making the case Understand the program requirements Establish a board of directors or steering commijee Involve key stakeholders in important decisions Provide updates Show evidence that simula/on works Show other people s ROI, if allowed BE PATIENT!

52 People Who are the experts? Where do you find experts? How many people do you need to succeed?

53 Kirkpatrick Model

54 Four Level Model Level 1: Par/cipant Reac/on Level 2: Learning Level 3: Job Impact Level 4: Business Impact Trainingcheck.com. (n.d.). Kirkpatrick ROE model of training evalua/on. Retrieved from hjp://trainingcheck.com/training- evalua/on /kirkpatrick- ROE- model- of- training- evalua/on/

55 Level 1 REACTION: how the learners react to the learning process Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /~donclark/hrd/isd/kirkpatrick.html

56 Level 2 LEARNING: the extent to which the learners gain knowledge and skills Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /~donclark/hrd/isd/kirkpatrick.html

57 Level 3 BEHAVIOR: capability to perform the learned skills while on the job Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /~donclark/hrd/isd/kirkpatrick.html

58 Level 4 RESULTS: includes such items as monetary, efficiency, moral, safety, etc. Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /~donclark/hrd/isd/kirkpatrick.html

59 Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /~donclark/hrd/isd/kirkpatrick.html

60 Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /~donclark/hrd/isd/kirkpatrick.html

61 The heart of any model is understanding the basic principles

62 Return on Expecta>on

63 Main principles Business/[organiza/onal] objec/ves are seen as a star/ng point. ( Trainingcheck.com, n.d.) Return on Expecta/on (ROE) is key ( Trainingcheck.com, n.d.) Collec/ve efforts are needed throughout an [organiza/on] to achieve success. ( Trainingcheck.com, n.d.) Trainingcheck.com. (n.d.). Kirkpatrick ROE model of training evalua/on. Retrieved from hjp://trainingcheck.com/training- evalua/on /kirkpatrick- ROE- model- of- training- evalua/on/

64 Return on Expecta/on Proac/ve, business partnership approach that unified teams. Defines training as a contributor to key business results. Value defined by business stakeholders in coopera/on with training. Focus on comprehensive evidence and a compelling story of value. Easy to understand, flexible, and cost- effec/ve. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

65 The key is

66 Numeric data Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

67 Tes)monials Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

68 Stories Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

69 The combina)on of evidence shapes a story that appeals to and is understood by everyone. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

70 You must s/ll: 1. Iden/fy key stakeholders. 2. Create a cross- func/onal advisory board. 3. Develop a strategic ini/a/ve, and use an ROE process. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

71 Collect Data!

72 Define data points Why are you collec/ng data? What informa/on is relevant? Intermingled into Pa/ent Safety? Does data impact Pa/ent Safety? What do you want to learn from your data?

73 Important Data Points Some Important data points Financials Simula/on hours Par/cipant numbers Total number of simula/ons Down/me? Number of visitors?

74 Outcomes Importance of data. Where can it be used? How can it be used? When can it be used?

75 Monthly Log

76 Monthly Log

77 Monthly Log

78 Monthly Log

79 Month End Data

80 First Quarter - Raw

81 First Quarter - Graphed

82 Targets

83 How does this data help?

84 Collec>on & Storage

85 Data Collec/on & Storage Collec/on Surveys Video logs Simula/on Data Storage Excel (example = WakeMed s system) Access database FileMaker

86 Simula/on Log

87 Quan/ta/ve vs. Qualita/ve data Quan/ta/ve: of, rela/ng to, or expressible in terms of QUANTITY. Qualita/ve: of, rela/ng to, or involving QUALITY or kind. Why is the difference important? the focus is what makes the difference. Is your program about numbers or the service you provide? hjp:// webster.com/dic/onary/quan/ta/ve

88 Integra>on

89 Collec/ng data in classes Examples: Pre- & Post- tests Airway Management Courses Simula/on trigged by an event Trends in performance Instructor & Students Logging /me for simulated events What types of classes do you teach? How can you begin collec/ng data? Let s draw it out!!

90 Human Processes IV Therapy How many processes exist in star/ng an IV? Cogni/ve vs. Prac/cal processes Checklist? If we analyze each step, where do we see infec/ons develop? Do small things impact safety? Finance? You?

91 Can I make money?

92 Genera/ng Revenue Is it possible? Can I sell programs? How do I jus/fy selling simula/on, crea/ng ROI or ROE, and keeping up with my internal customers? What programs do I sell? What is required? Contracts? Legal concerns?

93 GeYng the word out Take home items (trinkets?) Pins Pens Folders Journal ar/cles, news stories Demonstra/ons Discussions Presenta/ons } a Return

94 A Return = Outcomes + Simula>on

95 The Link Clinical outcomes Variability of outcomes Transi/on to prac/ce Pa/ent safety Change in behavior as a result of educa/on Iden/fying cogni/ve and prac/cal gaps

96 Where can I calculate a return from? 1. System- wide process changes 2. Changes in programs as a result of simula/on 3. Increase in program demand 4. Iden/fica/on of gaps 5. The stories.

97 What is the difference between ROI & ROE?

98 ROE verse ROI Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://

99 Our Story

100

101

102

103

104

105

106

107

108

109

110

111

112 Overview ROI ROE Educa/on Principles & Kirkpatrick Data in simula/on courses Link simula/on to ROI Calcula/ons and examples Where can ROI be calculate from? ROE or ROI?

113 References Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp:// /thevalueof_it_investments.pdf Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp:// /isd/kirkpatrick.html Cooke, M., Irby, D., Sullivan, W., & Ludmerer, K. (2006). American medical educa/on 100 years a]er the Flexner report. The New England Journal of Medicine, 355(13), Entertainment So]ware Associa/on. (2010). Industry facts. Retrieved from hjp:// Investopedia. (n.d.). Return on investment - ROI. Retrieved from hjp:// Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp:// Markow, M. (2011). Business terms glossary. BusinessWings. Retrieved from hjp:// terms- glossary Page, D. (2010). IT s return on investments is tricky to pin down. Hospital & Health Networks. Retrieved from hjp:// /ar/cledisplay.jsp?dcrpath=hhnmag/ar/cle/data/06jun2010/1006hhn_fea_mostwired&domain=hhnmag Robert Wood Johnson Founda/on. (2011). Glossary of healthcare terms. Quality/Equality. Retrieved from hjp:// SBA.gov. (2011). Essen/al elements of a good business plan. Retrieved from hjp:// structure/star/ng- managing- business /star/ng- business/wri/ng- business- plan/essen/al- elements- good- busines Smith, R. (2009). Embracing game technology for medical educa/on. Retrieved from hjp:// 2009_RSmith_MT3.pdf Solu/on Matrix. (2011). Return on investment. Retrieved from hjp:// on- investment.html Trainingcheck.com. (n.d.). Kirkpatrick ROE model of training evalua/on. Retrieved from hjp://trainingcheck.com/training- evalua/on /kirkpatrick- ROE- model- of- training- evalua/on/

114 ANY QUESTIONS??