Using Data to Help improve a Return on Investment

Using Data to Help improve a Return on Investment Amar P. Patel, MS, NREMT-P, CFC Director Center for Innovative Learning WakeMed Health & Hospitals Raleigh, North Carolina

Overview ROI ROE Educa/onal Principles & Kirkpatrick Data in simula/on courses Link simula/on to ROI Calcula/ons and examples Where can ROI be calculate from? ROE or ROI?

Objec/ves Define Return of Investment (ROI). Describe how to calculate Return of Investment. Discuss how to link simula/on and ROI. Demonstrate an ROI calcula/on u/lizing informa/on provided through audience feedback. Determine 5 key areas ROI can be calculated from.

Questions? How do we utilize existing technology to integrate a multi-disciplinary group? How do we utilize simulation to teach multiple disciplines? What are these opportunities?

A simple understanding of history

Average Age 38 Entertainment Software Association. (2010). Industry facts. Retrieved from http://www.theesa.com/facts/index.asp

Growing Issues Medical procedures are becoming more numerous and more complex medical knowledge has hypertrophied. (Cooke, et al, 2006) Clinical rota/ons are becoming more difficult to schedule. Clinical seyng is not the place to prac/ce skills. Cooke, M., Irby, D., Sullivan, W., & Ludmerer, K. (2006). American medical educa/on 100 years a]er the flexner report. The New England Journal of Medicine, 355(13), 1339-1344.

What does simula/on offer? Repe//on which leads to proficiency Understanding of the cogni/ve and the prac/cal Training opportuni/es Established objec/ves Clear expecta/ons Quality & Quan/ty Immediate applica/on

Funding How do you secure funding? What does that mean? For what period of /me? Can a simula/on program be self- sufficient? What do investors want? Repor/ng on funding?

Consider these costs

Developing a sim center Are you looking to develop a new building or use an exis/ng building? A cost is based on program objec/ves? short & long term goals? your target audience? Why does it majer?

Investments Program investment = $1 million Student count = 1,000 people/year Cost per student = $1,000.00 What about intangibles? Long term expenses? Supplies? Gases? - - - What does that cost? What are the true financial needs?

Intangibles Community Outreach Dona/ons High visibility show you mean business! Does there need to be a business model? What is free and what costs money?

Is simula)on profitable?

Is it?

Mo/ve 1: Lower Cost } Powerful Mo>vators Mo/ve 2: Bejer Access to Symptoms/Cases Mo/ve 3: Reduced Training Time Mo/ve 4: Reduced Errors Smith, R. (2009). Embracing game technology for medical educa/on. Retrieved from hjp://www.peostri.army.mil/cto/files/2009_rsmith_mt3.pdf

Business Plan

Defined as: A wrijen document which describes the business, its objec/ves, its strategies, the market in which it operates and its financial forecasts. (Markow, 2011) Markow, M. (2011). Business terms glossary. BusinessWings. Retrieved from hjp://www.businesswings.co.uk/ar/cles/business- terms- glossary

Consists of: Business Plan Execu/ve Summary* Market Analysis Company Descrip/on Organiza/on & Management Marke/ng & Sales Management Service or Product Line Funding Request Financials Appendix SBA.gov. (2011). Essen/al elements of a good business plan. Retrieved from hjp://www.sba.gov/category/naviga/on- structure /star/ng- managing- business/star/ng- business/wri/ng- business- plan/essen/al- elements- good- busines

Remember this: hospitals typically measure ROI from a business perspec/ve cost, revenues or opera/ng efficiencies but many benefits of clinical applica/ons fall into quality and safety realms that do not easily translate into dollars. (Page, 2010) Page, D. (2010). IT s return on investments is tricky to pin down. Hospital & Health Networks. Retrieved from hjp://www.hhnmag.com/hhnmag_app/jsp /ar/cledisplay.jsp?dcrpath=hhnmag/ar/cle/data/06jun2010/1006hhn_fea_mostwired&domain=hhnmag

A business plan will only make you stronger

The reality is: "We should measure clinical ROI in terms of measurable impact on pa/ent care. (Page, 2010) Page, D. (2010). IT s return on investments is tricky to pin down. Hospital & Health Networks. Retrieved from hjp://www.hhnmag.com/hhnmag_app/jsp /ar/cledisplay.jsp?dcrpath=hhnmag/ar/cle/data/06jun2010/1006hhn_fea_mostwired&domain=hhnmag

A Return on Investment

Tradi/onal ROI is: performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. ( Investopedia, n.d.) if an investment does not have a posi/ve ROI, or if there are other opportuni/es with a higher ROI, then the investment should be not be undertaken. ( Investopedia, n.d.) Investopedia. (n.d.). Return on investment - ROI. Retrieved from hjp://www.investopedia.com/terms/r/returnoninvestment.asp

Solu/on Matrix. (2011). Return on investment. Retrieved from hjp://www.solu/onmatrix.com/return- on- investment.html

Tradi/onal ROI ROI evaluates an investment s poten/al by comparing the magnitude and /ming of expected gains to the investments costs. ( Best, 2002) ROI = gains investments costs Investments costs $700,000 (saved) $500,000 sim center $500,000 = 40% Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp://www.cio.gov/documents /thevalueof_it_investments.pdf

Healthcare ROI is: the amount of improvement in care brought about by a certain investment. ROI can also refer to the theory that if you invest in health care quality now, then the quality of care for pa/ents will improve in the future. ( Robert Wood, 2011) Robert Wood Johnson Founda/on. (2011). Glossary of healthcare terms. Quality/Equality. Retrieved from hjp://www.rwjf.org/qualityequality/glossary.jsp

Healthcare ROI 1. Examines impact to system processes. 2. Determines a cost to each process related change and if the cost was favorable. 3. That cost is consider the return on investment. ((# Process errors found x Es/mated cost per error)- SIM COSTS) x (% $ Paid Out for errors) SIM COSTS

Healthcare ROI ROI Total: 5.93% Number of Process Errors Found: 14 Es/mated Cost per Error: $35,000.00 } % Hospital $ Paid Out: 0.25% SIM Costs: $19,821.00 $490,000

Healthcare ROI Issues Who is actually providing the training that is resul/ng in the ROI being created? How are you determining process issues? Are there clear program objec/ves? How are you being funded? Who benefits from the ROI? Who are your stakeholders?

Return on Investment Defensive isola/on approach that separates func/ons. Defines training as an end in itself. Value defined by predetermined formula. Focus on single metric of numeric proof. Complex, rigid, and expensive. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

Let s try a few calcula)ons

To create ROI, we must develop a value measuring methodology. ( Best, 2002) Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp://www.cio.gov/documents /thevalueof_it_investments.pdf

Value Measuring Methodology 1. Develop a decision framework Customers Social Government Opera/onal/Founda/onal Strategic/Poli/cal Government Financial 2. Perform an alterna/ve analysis 3. Gather all the informa/on 4. Communicate and document Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp://www.cio.gov/documents /thevalueof_it_investments.pdf

What does all this mean? There has been this ongoing debate in medical simula/on about the ability to prove ones worth - then been able to u/lize that proof to secure addi/onal funding from donors. Does simula/on make an impact in educa/on? Does it make an impact in healthcare, in pa/ent safety, in changes in human behavior or system processes? There are numerous issues that result from trying to jus/fy that simula/on was the sole reason a system benefited. Was it the single cause or was there other educa/onal programs that helped shaped changes? What happens if the data shows decline? Was simula/on to blame?

The mo/va/on The want and desire to include simula/on. It s the in thing to do The data shows improvement in cogni/ve learning. The data shows improvement in prac/cal applica/on and reten/on.

But

You have to think through it!

Business side Who are your key stakeholders? Internal External Have you considered developing a business plan? Do you have opera/onal & strategic goals? Who is managing your program data & finances?

When can I see a return?

Making the case Understand the program requirements Establish a board of directors or steering commijee Involve key stakeholders in important decisions Provide updates Show evidence that simula/on works Show other people s ROI, if allowed BE PATIENT!

People Who are the experts? Where do you find experts? How many people do you need to succeed?

Kirkpatrick Model

Four Level Model Level 1: Par/cipant Reac/on Level 2: Learning Level 3: Job Impact Level 4: Business Impact Trainingcheck.com. (n.d.). Kirkpatrick ROE model of training evalua/on. Retrieved from hjp://trainingcheck.com/training- evalua/on /kirkpatrick- ROE- model- of- training- evalua/on/

Level 1 REACTION: how the learners react to the learning process Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com /~donclark/hrd/isd/kirkpatrick.html

Level 2 LEARNING: the extent to which the learners gain knowledge and skills Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com /~donclark/hrd/isd/kirkpatrick.html

Level 3 BEHAVIOR: capability to perform the learned skills while on the job Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com /~donclark/hrd/isd/kirkpatrick.html

Level 4 RESULTS: includes such items as monetary, efficiency, moral, safety, etc. Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com /~donclark/hrd/isd/kirkpatrick.html

Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com /~donclark/hrd/isd/kirkpatrick.html

Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com /~donclark/hrd/isd/kirkpatrick.html

The heart of any model is understanding the basic principles

Return on Expecta>on

Main principles Business/[organiza/onal] objec/ves are seen as a star/ng point. ( Trainingcheck.com, n.d.) Return on Expecta/on (ROE) is key ( Trainingcheck.com, n.d.) Collec/ve efforts are needed throughout an [organiza/on] to achieve success. ( Trainingcheck.com, n.d.) Trainingcheck.com. (n.d.). Kirkpatrick ROE model of training evalua/on. Retrieved from hjp://trainingcheck.com/training- evalua/on /kirkpatrick- ROE- model- of- training- evalua/on/

Return on Expecta/on Proac/ve, business partnership approach that unified teams. Defines training as a contributor to key business results. Value defined by business stakeholders in coopera/on with training. Focus on comprehensive evidence and a compelling story of value. Easy to understand, flexible, and cost- effec/ve. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

The key is

Numeric data Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

Tes)monials Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

Stories Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

The combina)on of evidence shapes a story that appeals to and is understood by everyone. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

You must s/ll: 1. Iden/fy key stakeholders. 2. Create a cross- func/onal advisory board. 3. Develop a strategic ini/a/ve, and use an ROE process. Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

Collect Data!

Define data points Why are you collec/ng data? What informa/on is relevant? Intermingled into Pa/ent Safety? Does data impact Pa/ent Safety? What do you want to learn from your data?

Important Data Points Some Important data points Financials Simula/on hours Par/cipant numbers Total number of simula/ons Down/me? Number of visitors?

Outcomes Importance of data. Where can it be used? How can it be used? When can it be used?

Monthly Log

Monthly Log

Monthly Log

Monthly Log

Month End Data

First Quarter - Raw

First Quarter - Graphed

Targets

How does this data help?

Collec>on & Storage

Data Collec/on & Storage Collec/on Surveys Video logs Simula/on Data Storage Excel (example = WakeMed s system) Access database FileMaker

Simula/on Log

Quan/ta/ve vs. Qualita/ve data Quan/ta/ve: of, rela/ng to, or expressible in terms of QUANTITY. Qualita/ve: of, rela/ng to, or involving QUALITY or kind. Why is the difference important? the focus is what makes the difference. Is your program about numbers or the service you provide? hjp://www.merriam- webster.com/dic/onary/quan/ta/ve

Integra>on

Collec/ng data in classes Examples: Pre- & Post- tests Airway Management Courses Simula/on trigged by an event Trends in performance Instructor & Students Logging /me for simulated events What types of classes do you teach? How can you begin collec/ng data? Let s draw it out!!

Human Processes IV Therapy How many processes exist in star/ng an IV? Cogni/ve vs. Prac/cal processes Checklist? If we analyze each step, where do we see infec/ons develop? Do small things impact safety? Finance? You?

Can I make money?

Genera/ng Revenue Is it possible? Can I sell programs? How do I jus/fy selling simula/on, crea/ng ROI or ROE, and keeping up with my internal customers? What programs do I sell? What is required? Contracts? Legal concerns?

GeYng the word out Take home items (trinkets?) Pins Pens Folders Journal ar/cles, news stories Demonstra/ons Discussions Presenta/ons } a Return

A Return = Outcomes + Simula>on

The Link Clinical outcomes Variability of outcomes Transi/on to prac/ce Pa/ent safety Change in behavior as a result of educa/on Iden/fying cogni/ve and prac/cal gaps

Where can I calculate a return from? 1. System- wide process changes 2. Changes in programs as a result of simula/on 3. Increase in program demand 4. Iden/fica/on of gaps 5. The stories.

What is the difference between ROI & ROE?

ROE verse ROI Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm

Our Story

Overview ROI ROE Educa/on Principles & Kirkpatrick Data in simula/on courses Link simula/on to ROI Calcula/ons and examples Where can ROI be calculate from? ROE or ROI?

References Best Prac/ce Commijee. (2002). The Value of IT investments: It s not just return on investments. Retrieved from hjp://www.cio.gov/documents /thevalueof_it_investments.pdf Big Dog & Lijle Dog s Performance Juxtaposi/on. (n.d.). Kirkpatrick s four- level training evalua/on model. Retrieved from hjp://www.nwlink.com/~donclark/hrd /isd/kirkpatrick.html Cooke, M., Irby, D., Sullivan, W., & Ludmerer, K. (2006). American medical educa/on 100 years a]er the Flexner report. The New England Journal of Medicine, 355(13), 1339-1344. Entertainment So]ware Associa/on. (2010). Industry facts. Retrieved from hjp://www.theesa.com/facts/index.asp Investopedia. (n.d.). Return on investment - ROI. Retrieved from hjp://www.investopedia.com/terms/r/returnoninvestment.asp Kirpatrick, JD, Kirkpatrick, WK. (2010). ROE s rising star: Why return on expecta/ons is geyng so much ajen/on. Training + Development. Retrieved from hjp://www.astd.org/td/archives/2010/aug/free/1008_roes_rising_star.htm Markow, M. (2011). Business terms glossary. BusinessWings. Retrieved from hjp://www.businesswings.co.uk/ar/cles/business- terms- glossary Page, D. (2010). IT s return on investments is tricky to pin down. Hospital & Health Networks. Retrieved from hjp://www.hhnmag.com/hhnmag_app/jsp /ar/cledisplay.jsp?dcrpath=hhnmag/ar/cle/data/06jun2010/1006hhn_fea_mostwired&domain=hhnmag Robert Wood Johnson Founda/on. (2011). Glossary of healthcare terms. Quality/Equality. Retrieved from hjp://www.rwjf.org/qualityequality/glossary.jsp SBA.gov. (2011). Essen/al elements of a good business plan. Retrieved from hjp://www.sba.gov/category/naviga/on- structure/star/ng- managing- business /star/ng- business/wri/ng- business- plan/essen/al- elements- good- busines Smith, R. (2009). Embracing game technology for medical educa/on. Retrieved from hjp://www.peostri.army.mil/cto/files 2009_RSmith_MT3.pdf Solu/on Matrix. (2011). Return on investment. Retrieved from hjp://www.solu/onmatrix.com/return- on- investment.html Trainingcheck.com. (n.d.). Kirkpatrick ROE model of training evalua/on. Retrieved from hjp://trainingcheck.com/training- evalua/on /kirkpatrick- ROE- model- of- training- evalua/on/

ANY QUESTIONS??