A Comprehensive Evaluation Framework for Telehealth Services NATHAN POULTNEY B.ICT (HONOURS), TELEHEALTH RESEARCH AND INNOVATION LABORATORY (THRIL)

Transcription

1 A Comprehensive Evaluation Framework for Telehealth Services NATHAN POULTNEY B.ICT (HONOURS), TELEHEALTH RESEARCH AND INNOVATION LABORATORY (THRIL) Telehealth is the delivery of health services and information remotely. Evaluation Framework is a formal structured approach to the evaluation of a service. Comprehensive in the sense that there is unifying themes across the range of evaluation techniques. SUPERVISORS: PROF. ANTHONY MAEDER, DR. JIM BASILAKIS Why do we need a Comprehensive Evaluation Framework for Telehealth? There is no universally accepted standard for evaluating a telehealth service, therefore all evaluations undertaken are done so differently. Current evaluation methodologies have varying weaknesses in what aspects are properly evaluated and what aspects are missed entirely. Many evaluation strategies focus on either clinical benefit or costbenefit, rarely both. Evaluations with emphasis on both often lack in performance in one of those areas. Literature Review Telehealth Projects in Australia Looking specifically at projects from the year 2000 onwards. With the objective of collating which projects show evidence of quality evaluation. Searches returned over 8,000 results many of which were not specific to telehealth evaluation within Australia. The papers that remained were culled down to 55 of interest. 22 of the papers were deemed acceptable in terms of the evaluation techniques adopted. The learnings from this literature review made it clear that a more comprehensive evaluation methodology is necessary in the area of Telehealth. Accepted Publications Publication Area of Health Study/ Summary Sample Size Evaluation/Analysis Methodology 1. Randomised controlled trial of an automated, interactive Diabetes Care and monitoring via automated telephone system 340 people Randomised Controlled telephone intervention to improve type 2 diabetes selfmanagement (Telephone-Linked Care Diabetes Project): study protocol. 2. Randomized controlled non-inferiority trial of a telehealth Speech Pathology Stuttering treatment via 40 people Randomised Controlled treatment for chronic stuttering: the Camperdown Program. Non-inferiority 3. Assessing exercise capacity using telehealth: a feasibility Cystic Fibrosis Physiological measurements via 10 people Feasibility Study study in adults with cystic fibrosis. 4. The feasibility of a community-based mobile telehealth Chronic eye and ear Screening patients in a vehicle and uploading results to 442 people of 760 that Feasibility Study screening service for Aboriginal and Torres Strait Islander diseases clinicians remotely consented children in Australia. 5. Young people's views on the potential use of telemedicine Sexual Health National survey on whether young people would have 662 people National Survey consultations for sexual health: results of a national survey. consultations regarding sexual health via telephone or 6. What's in a message? Delivering sexual health promotion to Sexual Health Sexual health promotion through SMS to young adults 43 people Evaluation Focus Groups young people in Australia via text messaging. to increase their likelihood of safe sex and regular checkups 7. How effective are short message service reminders at N/A Determination of whether SMS reminders improve 18 reports Meta-Analysis and increasing clinic attendance? A meta-analysis and systematic clinical attendance Systematic Review review. 8. A pilot trial of emergency telemedicine in regional Western Emergency Care Telemedicine (primarily ) assisting 25 sites (locations) Feasibility Study Australia. clinicians in remote diagnosis and care 9. Telemedicine for wound care : Current practice and future Wound Care Store-and-forward and real-time approaches to wound 1,300+ people Literature Review potential care, a literature review 10. Assessment of childrens' literacy via an Internet-based Literacy The assessment of children's literacy via 20 people Randomised Controlled telehealth system compared to face-to-face consultations 11. The successes and challenges of providing a paediatric burns Paediatric Burns Clinical reviews of paediatric burns via 30 people Survey service by telehealth in Western Australia Accepted Publications contd. Publication Area of Health Study/ Summary Sample Size Evaluation/Analysis Methodology 13. Successful treatment of patients with hepatitis C in rural Hepatitis C Patients in remote or rural areas were reviewed and 35 people of 50 completed Feasibility Study and and remote Western Australia via telehealth. treated for hepatitis C via surveys Survey 14. Does the use of store-and-forward telehealth systems Diabetic Foot Ulcers Store-and-forward approach used to report on the 8 people Survey improve outcomes for clinicians managing diabetic foot ulcers status of diabetic foot ulcers? A pilot study 15. The effectiveness of digital imaging and remote expert Wound Care Store-and-forward approach to wound care 93 people Randomised Controlled wound consultation on healing rates in chronic lower leg ulcers in the Kimberley region of Western Australia 16. Responding to mental health emergencies: implementation Mental Health 24-hour mental health specialists available via 558 people Mixed Methods Evaluation of an innovative telehealth service in rural and remote New to rural and remote populations South Wales, Australia. 17. Operational teledermatology in Broken Hill, rural Dermatology Dermatology diagnosis and treatment through the use 46 people Feasibility Study Australia. of digital images, and fax 18. A review of Medicare expenditure in Australia for Psychiatry Examination of the costs involved with telepsychiatry N/A Retrospective Review psychiatric consultations delivered in person and via videoconference. 19. The point-of-referral barrier--a factor in the success of N/A A full-time coordinator for handling consultation 110 people Feasibility Study telehealth requests 20. Home videophones improve direct observation in Tuberculosis Monitoring tuberculosis treatment via 128 people Mixed Methods Evaluation tuberculosis treatment: a mixed methods evaluation. 21. Internet-based telehealth assessment of language using the Speech-language Online based speech assessment and its effectiveness 25 people Feasibility Study CELF-4 Pathology over face-to-face assessments 22. A national survey of telehealth activity in Australian N/A National survey of Australian hospitals and their 564 hospitals of the 814 National Survey hospitals. uptake of telehealth contacted 12. The reported benefits of telehealth for rural Australians N/A Literature review looking for benefits in telehealth for remote and rural populations of Australia 143 references Literature Review 1

2 Literature Review Evaluation Landscape The purpose of the literature review was to review existing telehealth and ehealth evaluation strategies, methodologies, frameworks and approaches. Common themes and issues in evaluation were identified. The value of a new evaluation methodology for telehealth services was realised. Overview Literature review on Telehealth projects. Literature review on evaluation methodologies in Telehealth and ehealth. A Comprehensive framework was developed as a result. Elements of four existing foundational frameworks were combined. Two case studies were undertaken utilising the framework. The case studies served to validate it s applicability in telehealth service evaluations. Foundational Frameworks Technology Acceptance Model (TAM) Davis, F. D. (1989), "Perceived usefulness, perceived ease of use, and user acceptance of information technology", MIS Quarterly, 13(3): A generic information and communications technology (ICT) evaluation framework (CHEATS) Shaw, N. (2002). CHEATS : a generic information communication technology (ICT) evaluation framework.computers in biology and medicine, 32(3), pp A Unified Approach for the Evaluation of Telehealth Implementations in Australia (IBES) Dattakumar, A., Gray, K., Jury, S., Biggs, B., Maeder, A., Noble, D., Borda, A., Schulz, T. and Gasko, H. (2013). A Unified Approach for the Evaluation of Telehealth Implementations in Australia. Institute for a Broadband Enabled Society. Telemedicine Maturity Model (TMMM) van Dyk, L. (2013). The Development of a Telemedicine Service Maturity Model. Stellenbosch: Stellenbosch University. Technology Acceptance Model version 1 introduced in 1986, TAM 2 in 2000 and TAM 3 was proposed in 2008 TAM version 1 CHEATS a generic information communication technology (ICT) evaluation framework (2002) Clinical, Human and organisational, Educational, Administrative, Technical, Social. IBES (Institute for a Broadband Enabled Society) Report A Unified Approach for the Evaluation of Telehealth Implementations in Australia (2013) [Patient Control], [Clinician Quality of Care] [Organisation sustainability], [Technology capability/capacity] 2

3 TAM strong user evaluation approach to acceptance of a technology or service. CHEATS an ICT approach to telehealth service evaluation that also covers clinical, organisational and human factors. TMMM Telemedicine Maturity Model (2013) Strengths IBES strong approach to sustainability, scalability in terms of finances. Technology aspects are addressed as are clinical measures specific to the Australian health context. TMMM excellent process orientated evaluation approach on multiple levels of maturity of a telehealth service. TAM no clinical, organisational acknowledgement, criticised for perceived ease-of-use being a relatively poor indicator for acceptance. Criticisms CHEATS IBES very vague in some parts, overlap or misplacement of aspects addressed. human factors are not addressed, technology scope is small, organisational aspect is purely finance related. TMMM only minor human factor. Developing the Comprehensive Framework The following common themes from the literature were taken into consideration during the development phase: Technology is core in telehealth. Administrative work is essential for smooth operations. Clinical aspects are also essential. Human related aspects are often overlooked in telehealth service evaluations. Support and admin personnel should also be included in evaluations. A Three Dimensional Hybrid Approach to Evaluation The Comprehensive Evaluation Framework consists of the following three dimensions that attempt to summarise the current evaluation landscape: Factors Addressed: aspects of a telehealth service that are to be evaluated. Frameworks: a collection of four previously mentioned foundational frameworks to use as the method for an evaluation. Participants: the people involved in a telehealth service to be evaluated. Comprehensive Framework Model Example Participants Support Personnel: medical technicians, clerical staff, hospital services staff, lab staff, IT staff Patient: individual patients with single episode of care / ongoing repeated care; patients with supporters or family members; cohorts of similar patients; Clinician: GP, Specialist, Community Health Worker, Doctor, Nurse Management Personnel: Administration, Business Owners, Technology Providers, Regulation Bodies, Third Party Financers, Health care Organisations 3

4 Case Study A The University of Melbourne Case Study A consisted of an interview with a doctor using a telehealth service based around video conferencing with migrants and asylum seekers. Followed by an open discussion with a professor from The University of Melbourne. Case Study A The interview with the doctor was in regards to his experiences using the telehealth service. Did he feel there were any challenges or difficulties in doing so. The open discussion was about the appropriateness, usefulness and value of the questions asked in the interview with the doctor and the answers received as a result, to serve as a form of validation. Case Study B Federation University Australia Case Study B consisted of an interview with a nurse using video conferencing software to undertake consults with a patient at her end and an oncologist at the other end. An open discussion was conducted with a professor from Federation University Australia. Case Study B The interview with the nurse was in regards to how well she perceives the technology to improve her ability to perform consults both independently and with the oncologist. The open discussion was once again about the appropriateness, usefulness and value of the questions asked in the interview with the nurse and the answers received as a result to serve as a form of validation. Results Case Study A Health Interview Professional Doctor It prevents travel and people in rural areas get seen more regularly i.e. every 3 months instead of 6 months. The picture can be poor due to bandwidth but sound is not an issue. One exception is when using an interpreter in a telehealth consultation as feedback from the interpreter is via phone. The quality is a little less but I think discomfort is minimal. Professor It's quite good because it's quite detailed. He's not just speaking in generalisations or hypothesizing. He's talking about his act in practical hands on experience of doing it. Which I think is the main value his information adds. Results Case Study B Health Interview Professional Nurse Patient s feel a bit of discomfort about talking to a TV. Younger patients are more accepting of talking to a screen. I had a couple of training sessions. I m still learning, it s a new system for many people. Advantages are being at a distance we have more access to the oncologist. Professor Consultations generally tend to be minutes, short. The tele presence gives a sense of being there, being in the same room as the patient, seeing if they can still communicate professionally and not be impeded on by the technology. I would say the questions are pretty relevant after reflecting on it. 4

5 Analysis Case Study A Telehealth is an effective method of dealing with migrant and asylum seeker patients. Issues highlighted from the interview are that of bandwidth, picture quality and occasional microphone feedback when an interpreter is present. Case Study B It has been found that the telehealth service used by the nurse and oncologist does allow for more patients to be seen by the specialist and reduces travel, costs and stress. Subsequently less patients would be able to see the specialist if they (or the specialist) have to travel for a consultation. Conclusions & Recommendations The case studies demonstrate how the Comprehensive Evaluation Framework transcends the foundational frameworks. Questions generated for the interviews are likely not possible with just one framework alone. If Questions were from just one foundational framework then it probably would not have covered the entire scope of the project as effectively. Further work would be valuable in applying more comprehensive evaluations in order to test the scope of the comprehensive framework across more of the factors addressed and participant group dimensions. Future Work As new frameworks or evaluation strategies are developed they too could benefit from being added to the comprehensive framework in order to see how well it stacks up against existing frameworks and evaluation methodologies. My Future Plans in Research Interested in pursuing a doctorate with the University of Western Sydney s THRIL group in the area of Evidence Based Practice in the telehealth service context. The honours project is good foundation work for moving into such an area that is similar to evaluation although less subjective and more scientific based. Thank you! References Davis, F. D. (1989), "Perceived usefulness, perceived ease of use, and user acceptance of information technology", MIS Quarterly, 13(3): Dattakumar, A., Gray, K., Jury, S., Biggs, B., Maeder, A., Noble, D., Borda, A., Schulz, T. and Gasko, H. (2013). A Unified Approach for the Evaluation of Telehealth Implementations in Australia. Institute for a Broadband Enabled Society. Legris, P., Ingham, J. and Collerette, P. (2003). Why do people use information technology? A critical review of the technology acceptance model. Information \& management, 40(3), pp Shaw, N. (2002). CHEATS : a generic information communication technology (ICT) evaluation framework.computers in biology and medicine, 32(3), pp Van Dyk, L. (2014). A Review of Telehealth Service Implementation Frameworks. International journal of environmental research and public health, 11(2), pp