Keywords: ehealth, health technology, evaluation framework, health technology assessment, health informatics evaluation

Transcription

1 EXPLORING HEALTH TECHNOLOGY EVALUATION IN THE CONTEXT OF EHEALTH: A QUALITATIVE STUDY TO UNDERSTAND ITS ROLE IN ENABLING INTERDISCIPLI- NARY HEALTHCARE Yu, Dan, The University of Sydney, School of Information Technologies, NSW 2006, Australia, dan.yu@sydney.edu.au Poon, Simon K., The University of Sydney, School of Information Technologies, NSW 2006, Australia, simon.poon@sydney.edu.au Tran, Vivienne, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, vtra7318@sydney.edu.au Lam, Mary, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, mary.lam@sydney.edu.au Hines, Monique, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, Monique.hines@sydney.edu.au Brunner, Melissa, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, Melissa.brunner@sydney.edu.au Lam, Mary, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, mary.lam@sydney.edu.au Keep, Melanie, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, Melanie.keep@sydney.edu.au Power, Emma, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, emma.power@sydney.edu.au Lowe, Robyn, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, robyn,lowe@sydney.edu.au Shaw, Tim, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, tim.shaw@sydney.edu.au Togher, Leanne, The University of Sydney, Faculty of Health Sciences, NSW 2141, Australia, leanne.togher@sydney.edu.au Abstract The complex relations between Health Technologies and clinical practices have been the focus of intensive research in recent years. There appears to be a discernible trend toward a more nuanced view in which the effects of the various clinical practices and their complex interactions with new health technologies are systematically evaluated. There is also emerging trend focusing on organisational factors and a greater shift towards a holistic view where evaluation of health technologies is linked to organisational practices. In this paper, we address the gaps in existing literature regarding the holistic evaluation of ehealth in clinical practice. We provide a synthesis of the existing evaluation frameworks for health technologies Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey, 2016

2 and report the results from a qualitative study conducted to gain insight into ehealth in practice within an interdisciplinary healthcare domain. Based on this synthesis and qualitative study, a conceptual framework is proposed for the evaluation of ehealth. The proposed framework will provide the foundation for the creation of a generic measurement model that allows for the comparative analysis of health technologies and assist in the decision-making of its stakeholders. Keywords: ehealth, health technology, evaluation framework, health technology assessment, health informatics evaluation Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

3 1 Introduction The growth of information technology (IT) in recent years has resulted in the rapid development of health technologies such as pharmaceutical drugs, medical devices, and ehealth applications. As health technologies move towards the development of more healthcare services, there is the need for systematic evaluation methods that determine the benefits, value, and costs of these technologies. Existing frameworks for the evaluation of health technologies are either derived from generic information systems evaluation methods such as the Technology Acceptance Model (Davis, 1989), or created for the purpose of evaluating technology within the medical domain, for example Health Technology Assessment (World Health Organisation, 2015). While these differing methods all aim to aid stakeholders decision-making and improve population health, differences in the evaluation criteria make it difficult to compare technologies and assess the true benefits of individual technologies. These evaluation methods tend to focus on understanding the safety, effectiveness, and usefulness of the health technology, and are no longer adequate to address the complexities that come with implementing ehealth technologies. The focus of this paper is to identify gaps within literature and provide insight into factors that need to be considered in the evaluation of ehealth. While this research aims to achieve an overarching understanding of health technology evaluation, there is a particular focus on ehealth and its application to healthcare, specifically traumatic brain injury (TBI) rehabilitation. We present results from a qualitative study that used focus groups to gain an understanding of clinician perspectives on the use of and issues related to ehealth in practice. The study provides insights into key issues regarding ehealth and facilitates the creation of a foundational conceptual framework for the evaluation of ehealth. This paper presents the steps of developing a conceptual model for evaluating the readiness of implementation of ehealth from the digital health perspective. 2 Review of Health Technology Evaluation While IT has the potential to improve health services, the complexity of health presents challenges to the implementation of effective health technologies; poorly designed health technologies may come at the cost of human lives. Thus, rigorous tools and techniques are required to ensure that the safety and effectiveness of new technologies are adequately evaluated. There are two distinct perspectives of evaluation frameworks and methods for health technologies throughout literature: health technology assessment and health informatics evaluation. The former is mostly used to evaluate pharmaceutical drugs and medical devices; the latter focusses on health informatics and health information systems. 2.1 Health Technology Assessment Health Technology Assessment involves the evaluation of health technologies in regards to its effectiveness and cost. Two prominent evaluation frameworks used internationally are Health Technology Assessment (HTA) and Comparative Effectiveness Research (CER). HTA is a health technology evaluation framework widely used to assist in the decision making and creation of health policy. In particular, HTA focuses on aiding governments and organisations in pricing and coverage decisions (Luce et al., 2010; Sorenson and Chalkidou, 2012). The World Health Organisation (WHO) broadly defines HTA as the "systematic evaluation of properties, effects, and impacts of health technology" (World Health Organisation, 2015). HTA considers the safety, clinical effectiveness, cost, organisational implications, social, legal and ethical considerations of new health technologies (Gagnon and Scott, 2005; Stephens et al., 2012; Hao and Thomas, 2013). Through examination of these factors, HTA is able to provide evidence on the safety, efficacy, and effectiveness of health technologies and practices in healthcare services. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

4 Although the definition of HTA encompasses a range of technologies including health information systems, it is predominantly used in the evaluation of pharmaceutical drugs and medical devices, with little or no focus on health informatics. Literature (Fairbrother et al., 2014; Gagnon and Scott, 2005; Nykänen et al., 2011) suggests that HTA does not adequately address the quality and safety of service delivery, which prevents its easy application to service related health technologies such as health informatics and ehealth. Another health technology evaluation framework is CER The Institute of Medicine defines CER as "the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care" (Institute of Medicine, 2009). The definition and goals of CER are comparable to that of HTA, where both methods assess the effectiveness of technology to provide evidence for stakeholders to be able to make more informed decisions (Fairbrother et al., 2014; Sox, 2010). Similar to HTA, the majority of CER cases focus on comparing pharmaceutical drugs and medical devices, with little evaluation of health informatics (Garber and Tunis, 2009; Giuliano et al., 2012). Across literature, it seems many countries use the terms HTA and CER interchangeably to address the evaluation of the effectiveness of medical technology (Fairbrother et al., 2014). The former is frequently used within Europe and Australia, while the latter is more commonly referred to in the United States (Fairbrother et al., 2014). Although similarities exist between the definitions of HTA and CER, a closer inspection of literature (Luce et al., 2010; Fairbrother et al., 2014; Giuliano et al., 2012) shows that there are slight differences in the factors emphasised within each framework. HTA involves the synthesis of existing evidence to assess on the clinical comparative effectiveness and cost effectiveness of an intervention CER compares the effectiveness of alternative technologies or services for a particular population. Furthermore, HTA focuses more heavily on the cost benefit assessment of technology, whereas CER emphasises comparative effectiveness. 2.2 Health Informatics Evaluation The second category of frameworks for evaluating health technologies is Health Informatics Evaluation, which focuses on the evaluation of health informatics and information systems by incorporating factors from users and organisational contexts. Numerous evaluation frameworks from information systems literature have been adapted to suit the health domain and healthcare systems. These include technology based models, user focused models, and organisational maturity and readiness models. To a large extent, health informatics evaluation stems from two established information system (IS) evaluation frameworks, DeLone and McLean s IS success model (1992) and the Technology Acceptance Model (TAM) (Davis, 1989). These generic frameworks have been used as a basis for the development of a number of proposed frameworks and guidelines for the specific evaluation of health informatics. DeLone and McLean s IS success model is a comprehensive model that captures the elements contributing to an information system s success (DeLone and McLean, 2003). This model heavily focuses on the technological effectiveness and quality and disregards human and external factors. Thus alone, this model may not be suitable for the evaluation of technologies within healthcare where the impact of technologies plays a large role in the user s life. Frameworks have been proposed that use the IS success model as a basis for evaluation, with the inclusion of other factors relevant to the health domain. One such framework is HOT-fit, which combines the system and technical factors of the IS success model with human and organisational factors in the evaluation of health information systems (Yusof et al., 2008). TAM is a user focused framework widely used to determine the factors that shape the use and acceptance of IT within an organisation (Davis, 1989). TAM s ability to capture users attitudes towards technology means that it may be appropriate for use in the healthcare context. However, the generic Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

5 nature of TAM and its emphasis on technological factors may fail to assess factors such as organisational processes and the environment of the system both crucial to health practice. Another set of important models drawn from a technical and organisational backgrounds are maturity and readiness models. The introduction of new technology in an organisation generally results in changes in processes and structure and involves the adoption of technical capabilities (Loureiro et al., 2007). These models provide an organisational perspective on the relationship between an organisation s state and technology adoption and implementation. The maturity model is used to assist the systematic development of organisational skills, processes and structures (Gresse von Wangenheim et al., 2013). In recent years, health organisations are increasingly adopting maturity models such as the HIMSS Continuity of Care Maturity Model, to help evaluate their status and improve their processes (de Carvalho et al., 2015; HIMSS Analytics, 2014). Maturity models provide organisations with a tool to address challenges in a structured way while also offering a reference point for the evaluation of their capabilities and a guide to improving them (de Carvalho et al., 2015). Although maturity models are generic, they should be considered as they can assist in understanding whether an organisation is fit for the adoption of a particular technology. Similarly, technology readiness models focus on organisational factors in technology acceptance. Readiness has been defined as "the level of fit between the IT innovation and the organisation" (Snyder-Halpern, 2001). Individuals need to feel confident and comfortable with new technology or their morale and productivity may decline. Thus, understanding individual reactions and attitudes towards technology is critical in determining the success of technology implementation within an organisation. Technology readiness models can provide insight into whether an organisation is willing and able to adopt new technology. Although there are only a few cases where this model has been applied in the healthcare context (Korst et al., 2011), it is still important to consider the organisation s readiness prior to technology implementation to determine which organisations it would suit best. 2.3 Summary Developments in IT have impacted the health industry greatly and carry with them the potential to improve the quality of healthcare. Although new interventions are constantly being developed and released, it is difficult to determine the true impact of these interventions due to the lack of a common comparative evaluation method across all health technologies. While some frameworks are tailored to the evaluation of pharmaceutical drugs and medical devices, others are more suitable for the evaluation of health informatics and information systems. Although health technology evaluation provides a rigorous means of evaluation, its emphasis on cost-benefit analyses limits its scope in health where there are other indicators of effectiveness. On the other hand, health informatics evaluation is more aligned with ehealth technologies in that it considers organisational, user and technical factors. However, these frameworks lack the rigorous methods employed in health technology evaluation. 3 Approaches to Evaluate ehealth Technology WHO formally defines ehealth as "the cost effective and secure use of information and communications technologies in support of health and health related fields" (World Health Organisation, 2004). While there is a general understanding of ehealth within the health industry, the exact definition of ehealth varies among users. Some (Pagliari et al., 2005) use the term synonymously with health informatics, telehealth, and mobile health, while others (Pawar et al., 2012; van Dyk, 2014) consider them to be distinct from ehealth, or view ehealth as an umbrella term that captures all these elements. This paper utilises the WHO definition of ehealth, which encompasses the various different elements such as health informatics, telehealth and mobile health. ehealth evaluation serves various purposes including the determination of its effectiveness, identification of unintended effects and consequences, and support for the creation of healthcare policy (Khoja Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

6 et al., 2012; Rahadhan et al., 2012). In contrast to medical technologies such as pharmaceutical drugs where there are formal frameworks in place for evaluation, there is a lack of standard evaluation methods for technologies for ehealth. This absence limits the opportunity to examine the quality, value and benefits of ehealth through comparative analysis. A strong evidence base for ehealth and its impact on financial and clinical outcomes is important for ensuring safe and effective technologyenabled healthcare. Hence, it is important that an evaluation framework is implemented to determine the impact of ehealth and to allow for comparative assessments. ehealth is broadly assessed through comparison of the intervention against a standard of outcome measures. Measures generally reflect a stakeholder s satisfaction with the intervention, the improvements in health and quality of the intervention (Suggs et al., 2006), however the assessment must ultimately consider each stakeholder involved and the input and outputs of the system. Furthermore, as ehealth is different to generic information systems, medical devices and pharmaceutical drugs, it presents unique challenges that need to be taken into account during evaluation. These challenges include the timing and indirectness of the impact of ehealth. The impact of ehealth on an individual s health is not immediate, thus there is a time lag between service use and its impact (Goletsis and Chletsos, 2010). Therefore, a longer time period is required, associated with higher cost for ehealth evaluation compared to traditional evaluation of information systems. Evaluation itself is difficult and with the added complexities that come with ehealth, it becomes more difficult to establish a holistic evaluation framework. A number of frameworks have been proposed for the evaluation of ehealth. While some frameworks provide evaluation guidelines of specific aspects of health information such as its quality, others provide evaluation criteria that encompass a range of aspects such as costs, benefits, and technical performance of the intervention (van Gemert-Pijnen et al., 2011; AHIMA, 2012). Catwell and Sheikh (2009) proposed a framework involving high level criteria and focused on understanding the relationships between environmental variables as part of the evaluation of ehealth. Others such as Hamid and Sarmad (2008) focused on the user s perspective such as the costs and benefits experienced by using the system. Alternatively, the framework proposed by the American Health Information Management Association (AHIMA) is solely for the purpose of providing a means for the evaluation of information quality (AHIMA, 2012). Although all frameworks share similar factors, there is not one which encompasses all factors that may be important for the evaluation of ehealth. There is a general concern that in spite of developments in ehealth, there is a lack of corresponding effective evaluation methodologies. It has been suggested (Ahern et al., 2006; Catwell and Sheikh, 2009; van Gemert-Pijnen et al., 2011) that ehealth should be evaluated in a similar manner to traditional medical treatment regimes such as pharmaceutical drugs, where rigorous evaluation approaches can assess the value added of ehealth. Therefore, it is important to consider the health technology evaluation methods of HTA and CER as a basis for ehealth evaluation. Nonetheless, the current lack of a single evaluation method for ehealth has led to difficulties in comparing ehealth applications against traditional healthcare methods. As evaluation is critical for the integration and sustainability of ehealth applications, there is a need for an agreed framework with common outcomes and measures. By understanding the different evaluation frameworks currently implemented, both generic and specific to health technologies, the benefits and limitations of these models can be determined. From these insights, a comprehensive framework can be formed to allow for the evaluation of all health technologies, including those that border the health and IT contexts, such as ehealth. 4 Methodology A qualitative study involving focus groups and in-depth interviews was conducted. The goal of this research was to gain insight and an in-depth understanding of clinician perspectives of using ehealth technologies as an interdisciplinary team and their attitudes towards the use of ehealth within the Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

7 workplace. This paper reports on data collected from one focus group conducted within a larger research study that comprised a series of focus groups and interviews in different workplaces. At the time of this report, only one of the focus groups had been conducted, and forms the basis of our initial analysis and findings in this paper. Focus group methods were based on previously established principles (Kitzinger, 1995) and were chosen to explore the shared perceptions on the concepts and attitudes towards using ehealth in interdisciplinary clinical practice (Kitzinger, 1994), as well as areas of controversy amongst participants. 4.1 Participant recruitment Traumatic Brain Injury (TBI) was chosen as the domain due to the interdisciplinary nature of the work and the potential benefits that ehealth can bring to such work (e.g. Togher et al., 2014). Thus, a focus group was conducted to gain an understanding of the attitudes surrounding ehealth and interdisciplinary work within rehabilitation for people after TBI. Participants consisted of healthcare professionals working in TBI rehabilitation in rural New South Wales (Australia), with recruitment based upon the purposeful sampling technique. 4.2 Data collection Four key questions were used to facilitate discussion between participants. The questions presented to participants were open-ended and aimed to elicit discussion that uncovered the participants experiences with ehealth, the challenges they faced, and their attitudes towards the use of ehealth in interdisciplinary healthcare. Additional sub-questions and visual prompts were provided as required to ensure that the discussion among the participants was relevant to the research questions. Four researchers assisted with facilitating the focus group and collecting data: the primary moderator, assistant moderator, and two assistants. All four researchers took notes on the discussion, and the discussion was digitally recorded and later transcribed verbatim. The transcript was de-identified to ensure anonymity of participants and organisations. The transcript was checked for accuracy against the digital recording by the researchers. The final transcript and a summary of key points were then ed to participants for verification (member checking). No request for alteration to the transcript or summary was made by any participant. 4.3 Data analysis A traditional approach was undertaken for qualitative analysis of the focus group transcript and is based upon the simple analysis framework by Krueger and Casey (2009). This involved the three main steps of: categorisation of raw data, descriptive statements and interpretation of data. As part of the first step, two researchers independently analysed and categorised data into categories. The categories of raw data were then cross-analysed in what way? and discussed by a group of researchers to ensure consensus and reliability in the development of descriptive statements for each theme. Once consensus was achieved on the themes and categories of the data, these descriptive statements were interpreted. As the analysis of focus group data involves a level of subjectivity, care was taken during the analysis to ensure that bias was not introduced in the interpretation of the results. We took into consideration any researcher preconceptions, the specific words spoken by the participants and the context of their statements, and the internal consistency and specificity of statements. To reduce subjectivity, findings were strengthened by having two researchers analyse the data independently, with the results and main findings then given to an independent reviewer for validation. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

8 5 Summary of data Five employees from a rurally based organisation in New South Wales (Australia) participated in the focus group. This organisation provides rehabilitation services to the community with the aim of helping clients transition back to their normal lives. The participants came from various healthcare backgrounds and were currently working in the same rehabilitation team that provides services to people with TBI. Factor Region Organisation type Services provided Disciplines of the health care professionals Focus group Rural New South Wales (Australia) 400km from the city of Sydney Non-government organisation Public inpatient and outpatient rehabilitation Speech pathology Occupational therapy Medical Nursing Care coordinator Years working in the team One over 10 years One 5-10 years Three 1-5 years Table 1. Summary of participant demographics 6 Results and analysis Six main themes emerged from data analysis which are individually discussed below. 6.1 Organisational structure Interdisciplinary healthcare generally involves a team of clinicians with expertise across a number of disciplines and co-located at their respective organisations. The variety of services and specialties located at one site allows for effective communication and teamwork and ease of access to a range of specialist skills within the team. The participants expressed a positive attitude towards being located at one site as it allows for effective information flows and easier communication between team members. Co-location of the team was often described as allowing strong relationships to be developed within the team. Robust team dynamics and a positive and strong team base were demonstrated by the interactions between the team members. The team also expressed positive attitudes towards working well face-to-face, leading to a period where they can improve processes and be proactive in using and integrating ehealth and technology into their workflow processes. They believe that they are reaching a point "where we ll be able to really be really proactive, and really improve things" through the use of ehealth. 6.2 Culture and attitudes towards technology The participants were all self-assessed as non-technology oriented people. Despite their lack of a technological background, the participants generally presented a positive attitude towards the use of Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

9 ehealth within their roles. While "there are certainly a number of people have embraced the technology really well and so they re really keen", others were more hesitant in embracing technology. The participants presented a positive attitude towards the use of ehealth within the workplace. The team perceived their situation as "lucky" and were grateful for having technology and demonstrated a positive attitude towards using technology in supporting their work. Despite some expressed concerns in relation to the use of ehealth, participants described a positive team culture towards improving how they work through the use of technology stating that with technology: "We could be magnificent. There s huge scope". This attitude suggests that the clinicians have an understanding of the potential benefits technology can bring to the team. While the team presented a positive attitude towards technology, they also expressed their concern that their ignorance of what could be done in regards to technology hinders their potential to find new and innovative solutions. 6.3 External organisations The team interacts with various external organisations including government organisations, hospitals and private practitioners. In fact, funding of services for people with TBI may bring together clinicians from a variety of organisations to provide interdisciplinary care for a client. Furthermore, clinicians frequently interact with the organisations previously involved in the care of their clients and who refer patients to the teams, such as acute hospitals, or metropolitan brain injury rehabilitation units. Lack of access to the electronic client records held by other organisations involved in client care means that clinicians rely heavily on traditional communication methods, such as telephone, to access client information. Such communication is variable in its quality and efficiency. Timely, efficient communication is generally experienced with other TBI unit services rather than generalist rehabilitation services as effective communication is fostered through a common understanding of TBI service delivery and clinicians information needs. The development of external provider knowledge together with an understanding of TBI services allows for effective interdisciplinary teamwork and communication between organisations. The participants have a positive perception of working with the external organisations the team has ongoing relationships with. For example, the team prefers to work with private providers with whom they already have developed a relationship, as "if we work with them most often it s easier. We tend to go back to a few of the same". However, differences in communication methods utilised by each organisation can hinder effective communication. For example, one government organisation uses mail or phone to communicate, while another government organisation only uses . This can cause inefficiencies for the clinicians, as they need to have an understanding of which communication method is appropriate for an external organisation. 6.4 IT support The level of support provided by the IT specialists varied due to having IT support located off-site in Sydney. Not being co-located with IT support makes it difficult for clinicians to communicate IT issues effectively over the phone or . Misunderstandings occurred between the two parties, as the IT staff "didn t really understand what we wanted, what we needed". The problem is further accentuated by clinicians having "difficulty in actually knowing the tech talk to know what to ask or know what the options [are]". Although this is a concern for the participants, they demonstrated a positive attitude in wanting to improve their understanding in IT, to minimise the technical expertise gap, stating "we d be keen to learn that tech talk and know what to go for". Furthermore, the change management process in obtaining sufficient support is long and complex, and often there was no response from IT specialists. The participants needed to continuously ask for something to be completed: "we just keep asking and asking". In addition to communication challenges, the long and complex process in gaining access is a barrier to effective use of the system to deliver services to clients. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

10 6.5 Technology, facilities and infrastructure As part of the study, the participants provided insight into the types of technologies and systems currently used within the team to support their roles. Various ehealth technologies and systems are used, ranging from basic technologies such as phones and s to more sophisticated systems. ehealth is utilised for communication, administrative tasks, clinical processes, client management and as part of client therapy and treatment. Client management is a core component for the organisation and electronic health records are one of the key systems used to support it. The team uses a number of different electronic health record systems to access client information. Although these systems share a purpose and perform similar functions, each provides varying information, resulting in duplication of data and inconsistencies in the information shared across the team. Moreover, a lack of interoperability between systems makes it difficult to transfer and share information smoothly. Interoperability of systems and the provision of information through a common access point, particularly for cases where interdisciplinary teams span multiple organisations, will allow for more effective and efficient service delivery and workflow processes. Having a single system used consistently in all organisations would improve communication and sharing of information, and would be easier for clinicians to learn and use Availability of IT and facilities The availability of technology and facilities to support the use of technology can be a barrier to effective work processes. For example, when using technology such as video conferencing, the technology and support facilities must be available and accessible at both communication sites. However, this is not always possible and coordinating access to such technology can be a complex task. For example, there is no video conferencing facilities onsite for the team to use, so they need to go to another community venue that provides the technology. The venue is not always accessible for clients and, as such, is not suitable to be used for client-based services. When these situations occur, the teams generally make do with the facilities that are available or revert to traditional, easily accessible communication technologies such as telephones. One participant described the situation as sufficient, but believed that the team could achieve "not just a great job, but a magnificent job, if we had more", suggesting that if technology and the respective facilities are available, it would provide the means for the clinicians to perform better Lack of basic infrastructure Many technologies and systems utilised by the team rely on the internet and the presence of a strong network connection. Thus, internet speed, high quality connections and accessibility to networking infrastructure are critical factors. Variations in internet quality and speed are hindering the effective use of technology and systems within the organisation. Furthermore, in rural regions, there are blackspots and 4G network restrictions due to the lack of infrastructure in place to support these network services. As such, the location of the organisation in rural NSW together with the lack of infrastructure creates an extra hindrance to the effective provision of services and use of technology. The team attempts to work within their limitations but without basic infrastructure in place they are unable to use even the most basic systems that can enable effective service provision. The participants strongly believe the lack of infrastructure, rather than the attitudes of the clinicians, is a barrier to working with technology and drives the team back to traditional means of performing processes: "so it s not the therapists being the barrier to progression and moving with the times, it s we don t have those infrastructures to try and do that." Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

11 6.6 Policies and guidelines The participants are required to adhere to various policies and guidelines provided by their organisation. However, there appears to be a lack of policies and a misalignment of internal policies with processes both internal and external to the organisation. Although clinicians are encouraged to utilise technology, there is a lack of guidelines on who can use particular technologies, which technologies can be used and how to use them. The internal policies and guidelines on the use of technology are insufficient and lack clarity. In general, individuals within the teams have restricted or no access to systems and technology such as Skype. The level of restriction varies from individual to individual, with some having full access and other having no access. This has resulted in confusion for staff due to inconsistencies in the level of access an individual has to a system. Moreover, simple actions such as sending text messages and ing photos are restricted and create barriers to the effective use of technology. There are situations where the number of restrictions on the technology makes it infeasible to use. For example, one participant requested a new smartphone in order to use it for communication. However, there were so many restrictions, such as the inability to download or use applications, and no access to the internet, that the device was unusable and as a result, the individual returned the device. The restrictive nature implies a lack of trust as part of the organisation culture, where employees are not trusted to work with IT as many applications and systems are restricted. The participants believe that there is a culture of treating the staff as children rather than professionals: "It s like instead of treating us like we re all professionals, professional people, you re treated like children who can t do that". There is a consensus that the restrictions need to be removed or a suitable level of access should be provided to clinicians to allow for effective and efficient work. Currently, the teams make do with what they have access to "make it work, but it s not the best". 7 Discussion This qualitative study brought to surface a number of key insights into the evaluation of ehealth. In particular, it highlighted the lack of a rigorous evaluation framework for non-traditional health technologies such as ehealth. Two main categories of evaluation methods for health technology are identified in the literature. One method, health technology assessment, focuses on the evaluation of health technology from a cost-benefit analysis perspective. These evaluation methods are generally utilised by government organisations to determine whether investment should be made for a health technology. As such, the evaluation is heavily driven by considerations of efficacy and cost effectiveness, which are most relevant in evaluation of traditional medical technologies such as pharmaceutical drugs and medical devices. In comparison, health informatics evaluation places emphasis on the evaluation of the technical, user and organisational factors. These methods are generally used to assess the fit of a health technology in its environment or context, rather than a population based evaluation of its benefits. Health information systems rarely have a single purpose or application, which makes it challenging to quantify or pinpoint the exact value the technology contributes to a single outcome. Furthermore, the majority of these frameworks are based on generic information systems evaluation techniques derived from sociology and organisational methodologies and studies, which may not necessarily achieve the level of rigour required for the evaluation of medical technologies. For health technologies that border the line of medical devices and health informatics such as ehealth, using either of these existing methods independently is unsuitable as these methods are unable to provide a holistic evaluation of such technology. The wide scope of ehealth creates challenges in the evaluation of such technologies, as they do not fit nicely within the scope of health technology or health informatics evaluation. The difficulty in quantifying the cost and benefits of these technologies make it challenging to evaluate under health technology evaluation methods such as HTA. However, if assessed under health informatics evaluation, there is a lack of standardised methods in the evaluation criteria. Organisational maturity models, such as the HIMSS Continuity of Care Maturity Model, at- Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

12 tempts to bridge the gap between health technology and health informatics evaluation by providing a means for the evaluation of technology and environment through the understanding of the organisation s capabilities. However, this model is aimed at health management and is not suitable for general health technology. Furthermore, there is currently no evaluation framework for ehealth that incorporates the critical components from both health technology and health informatics evaluation to provide a holistic understanding of the associated benefits and costs. This poses the question of whether it is possible to create an ehealth evaluation framework that incorporates the rigour of health technology evaluation with the flexibility and understanding of the technology s interaction with external factors achieved in health informatics evaluation techniques. The results from the qualitative study confirmed the issues raised in existing literature, including the need for rigorous evaluation of ehealth and the importance of considering the technology-environment fit. The findings also moved beyond the literature and offers new factors that need to be considered during the evaluation of ehealth. One of the most interesting findings from the study is that although the clinicians faced a number of challenges in regards to the use of ehealth and were unable to achieve the full benefits that it can offer, they presented positive attitudes towards ehealth. The majority of issues highlighted by the study are touched upon in literature, however the focus is still largely on the technological aspects of evaluation such as functionality and the quality of the system, rather than having an equal focus on the external and environmental impacts on technology. Without understanding organisational processes and policies, and having infrastructure in place, even if the technology is effective, it will not likely to be successfully implemented. Thus, factors from both health technology and health informatics evaluation should be drawn to provide the foundation for a rigorous ehealth evaluation model that considers both the effectiveness of technology and the fit of technology within its environment. 7.1 Proposal of a health technology evaluation model This study highlights the gaps within literature regarding the evaluation of ehealth within organisations. In particular, there is no evaluation model that considers the environment of health technology implementation and includes factors such as the user and organisation. The effect of this gap can be seen in the study, where ehealth was implemented with the belief that it can improve processes and client care, however these desired results were not observed. The results indicate that there is a lack of emphasis on particular elements in the evaluation of ehealth in literature such as infrastructure and policy. In the literature, many of these issues are often overlooked as being of lesser importance to factors such as technological and user impacts. We propose a foundational framework that aims to integrate the two evaluation processes and bridge the gap between the evaluation of the health technology itself and the evaluation of the fit between the technology and its environment. These two processes need to be integrated in order to determine whether the technology is effective and safe to use, as well as consider the type of organisation or individual that the technology is suited for. As a result, the proposed framework consists of five evaluation stages: external environment, health technology, user, organisation and governance. The model is depicted in Figure 1 and highlights the five evaluation stages following a continuous improvement process. The cyclical nature of the model emphasises the importance of continuous evaluation and improvement. As technology is continuously improving and developing, evaluation needs to occur simultaneously and evolve with the technology in order to determine the suitability of the technology in an ever-changing environment. Although the model is cyclical, the starting point of the evaluation should be the "External environment" stage as the health technology and its surrounding environment needs to be aligned before the health technology itself is evaluated. This is because the success of the technology relies upon having supporting foundations within the environment. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

13 Figure 1. Proposed framework for the evaluation of ehealth The first stage is evaluating the external environment. This step aims to determine whether the environment in which the ehealth technology is to be implemented will be suitable. This stage assesses the availability and quality of the infrastructure, facilities and support. The second stage involves the evaluation of the health technology itself and assesses the effectiveness, cost effectiveness, efficacy, safety and quality of the health technology. The main aim of this stage is to determine if the technology is effective in achieving its purpose. This may be determined through the evaluation of the user satisfaction and ease of use of the technology as part of understanding the quality of the health technology. The next two stages assess the fit between the technology, user, and organisation, and aim to determine whether the technology will be adopted by the users and organisation, and which users and organisations the technology will be suitable for. The final stage, governance, involves the evaluation of policies and guidelines that may be influenced by the technology. This stage is crucial to the evaluation as it ties all the other stages together. Once reached the technology should be considered, at a minimum, as having satisfactory meeting the criteria of the other components. However, as technology and the environment continue to develop, the evaluation of the health technology and its components should continue simultaneously. Thus, if required, further evaluation should occur by repeating the five stages until a satisfactory outcome is achieved. As this framework is foundational and only built from the findings from a single focus group, there are some limitations. These include the lack of detail about how this framework will be utilised and the exact data and constructs to be included as part of this evaluation framework. Further work, including conducting remaining focus groups and individual interviews as part of the larger study will shed light on these details for the framework. In comparison to existing frameworks, which focus on either the evaluation of ehealth or traditional health technologies, the proposed framework can potentially provide a way to evaluate all health technologies and thus provide a common means of evaluation of these technologies. Overall, this framework aims to provide a solution to the gap within literature, that is a framework that captures the rigour and cost benefit focus of health technology evaluation, and the technological, organisational and user factors involved in health informatics evaluation. Current literature indicates the challenges faced in the evaluation of non-traditional health technologies such as ehealth, however there is no coherent solution to bridging the gaps between the existing models. Thus, the proposed framework attempts to integrate health technology evaluation and health informatics evaluation to provide a means for the holistic evaluation of ehealth. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

14 7.2 Limitations of research A number of limitations exist regarding the study and the proposed evaluation framework. A small sample of the population was chosen for this study and as such, the findings from the study may not have reached the point of saturation where additional data collection does not provide any new insights or identified themes. Although other means were undertaken to ensure the trustworthiness of results, further qualitative research should be undertaken to ensure that there are no additional findings and insights to be gained. Furthermore, as is typical of qualitative studies, the results from the study cannot be generalised to other contexts. Additional perspectives from managers and other organisational staff such as IT staff can provide further insight into the topic and allow for refinement of the framework. Testing is yet to be conducted to determine the feasibility, validity and reliability of the framework. Currently, it is unknown whether the framework captures all the elements required in the evaluation of ehealth. The framework provides some initial insight into factors that should be considered during ehealth evaluation studies. Furthermore, the testing of the proposed framework will shed light on whether the introduction of a rigorous ehealth evaluation framework will lead to better evaluation of health technologies and understanding of how these technologies can be utilised in improving the healthcare domain. 8 Conclusion Health technologies have the potential to improve the quality of healthcare services. Literature has shown that although ehealth has many benefits, there is a lack of rigorous evaluation being conducted resulting in a lack evidence to support these claims. Due to lack of tools and methods to enable us to capture all the elements necessary for the holistic evaluation of health technologies, there is a critical gap in the evaluation of technologies that border the line of traditional medical technology and health informatics, such as ehealth. In this research, we present a qualitative study that provides insight into the evaluation of ehealth within the interdisciplinary TBI rehabilitation. From the results of the qualitative study and the synthesis of literature, we proposed an initial framework for the evaluation of ehealth. The proposed fivefactor model provides a framework to be further developed into a measurement model for evaluating ehealth technology in a systematic manner. Based on our findings, we have provided some initial basis for the determination of the factors that should be considered in evaluation studies, and these factors have potential for comparative analysis to be conducted across ehealth technologies. Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

15 References Ahern, D. K., Kreslake, J. M., and Phalen, J. M What is ehealth (6): Perspectives on the evolution of ehealth research. Journal of Medical Internet Research, 8 (1), e4. AHIMA Data quality management model (updated). Journal of AHIMA, 83 (7), Catwell, L. and Sheikh, A Evaluating ehealth interventions: The need for continuous systemic evaluation. PLoS Medicine, 6(8), e Davis, F. D Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), de Carvalho, J., Rocha, Á., and Vasconcelos, J Towards an encompassing maturity model for the management of hospital information systems. Journal of Medical Systems, 39(9), 1 9. DeLone, W. H. and McLean, E. R Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), DeLone, W. H. and McLean, E. R The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), Fairbrother, G., O Brien, E., Pradhananga, R., and Chalkidou, K Improving quality and efficiency in health care through comparative effectiveness analyses: An international perspective., AcademyHealth. Gagnon, M.-P. and Scott, R. E Striving for evidence in e-health evaluation: lessons from health technology assessment. Journal of Telemedicine and Telecare, 11:S34 6. Garber, A. M. and Tunis, Sean R, M Does comparative-effectiveness research threaten personalized medicine? The New England Journal of Medicine, 360(19), Geissbuhler, A How can ehealth help fix broken health systems? Methods of Information in Medicine, 50(4), Giuliano, K. K., Ferguson, M., and Silfen, E Medical technology innovation and the importance of comparative effectiveness research. Journal of Medical Marketing, 12(1), Goletsis, Y. and Chletsos, M Towards a unified methodology for the evaluation of e-health applications. In: Proceedings of the 10th IEEE International Conference on Information Technology and Applications in Biomedicine, pages 1 4. Gresse von Wangenheim, C., von Wangenheim, A., McCaffery, F., Hauck, J., and Buglione, L Tailoring software process capability/maturity models for the health domain. Health and Technology, 3(1), Hamid, A. and Sarmad, A Evaluation of e-health services: user s perspective criteria. Transforming Government: People, Process and Policy, 2(4), Hao, Y. and Thomas, A Health technology assessment and comparative effectiveness research: a pharmaceutical industry perspective. Expert Review of Pharmacoeconomics & Outcomes Research, 13(4), HIMSS Analytics Continuity of care maturity model. URL: (viewed 20/09/2015) Holden, R. J. and Karsh, B.-T The technology acceptance model: Its past and its future in health care. Journal of Biomedical Informatics, 43(1), Institute of Medicine Initial national priorities for comparative effectiveness research. Technical report, Institute of Medicine. Khoja, S., Durrani, H., Scott, R. E., Sajwani, A., and Piryani, U Conceptual framework for development of comprehensive e-health evaluation tool. Telemedicine and e-health, 19(1), Kitzinger, J The methodology of Focus Groups: the importance of interaction between research participants. Sociology of Health and Illness, 16(1), Kitzinger, J Qualitative research. Introducing focus groups. British Medical Journal, 311(7000), Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

16 Korst, L. M., Aydin, C. E., Signer, J. M. K., and Fink, A Hospital readiness for health information exchange: Development of metrics associated with successful collaboration for quality improvement. International Journal of Medical Informatics, 80(8), e178 e188. Krueger, R. A. and Casey, M. A Focus groups: a practical guide for applied research. Sage Publications, fourth edition. Lewis, L. M The promise and ambiguity of ehealth research. Nursing Research, 64(3), Loureiro, G., Curran, R., Zephir, O., and Minel, S Reaching readiness in technological change through the application of capability maturity models principals, Complex Systems Concurrent Engineering, pages Springer London. Luce, B. R., Drummond, M., Jönsson, B., Neumann, P. J., Schwartz, J. S., Siebert, U., and Sullivan, S. D EBM, HTA, and CER: Clearing the confusion. The Milbank Quarterly, 88(2), Nykänen, P., Brender, J., Talmon, J., de Keizer, N., Rigby, M., Beuscart-Zephir, M.-C., and Ammenwerth, E Guideline for good evaluation practice in health informatics (gep-hi). International journal of medical informatics, 80(12), Oh, H., Rizo, C., Enkin, M., and Jadad, A What is ehealth (3): A systematic review of published definitions. Journal of Medical Internet Research, 7(1), e1. Pagliari, C., Sloan, D., Gregor, P., Sullivan, F., Detmer, D., Kahan, J. P., Oortwijn, W., and MacGillivray, S What is ehealth (4): A scoping exercise to map the field. Journal of Medical Internet Research, 7(1), e9. Pawar, P., Jones, V., van Beijnum, B.-J. F., and Hermens, H A framework for the comparison of mobile patient monitoring systems. Journal of Biomedical Informatics, 45(3), Rahadhan, P., Poon, S.K. and Land, L Understanding Unintended Consequences for Electronic Medical Records (EMR): A Literature Review, In Proceedings of the 2012 Health Informatics Conference (HIC2012), Sydney, Australia, 30 July - 2 August. Snyder-Halpern, R Indicators of organizational readiness for clinical information technology/systems innovation: a Delphi study. International Journal of Medical Informatics, 63(3), Sorenson, C. and Chalkidou, K Reflections on the evolution of health technology assessment in Europe. Health Economics, Policy and Law, 7(1), Sox, H. C Defining comparative effectiveness research: The importance of getting it right. Medical care, 48(6 Suppl 1), S7 S8. Stephens, J. M., Handke, B., and Doshi, J. A International survey of methods used in health technology assessment (HTA): does practice meet the principles proposed for good research? Comparative Effectiveness Research, 2: Suggs, L. S., Cowdery, J. E., and Carroll, J. B Tailored program evaluation: Past, present, future. Evaluation and Program Planning, 29(4), Togher, L., Douglas, J., Teasell, R., and Turkstra, L.S INCOG Recommendations for Management of Cognition Following Traumatic Brain Injury, PART IV: Cognitive Communication. Journal of Head Trauma Rehabilitation, 29(4), van Dyk, L A review of telehealth service implementation frameworks. International Journal of Environmental Research and Public Health, 11(2), van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., and Seydel, E. R A holistic framework to improve the uptake and impact of ehealth technologies. Journal of Medical Internet Research, 13(4), e111. World Health Organisation ehealth strategy document presented to 115th session of executive board. Technical report, World Health Organisation. World Health Organisation Health technology assessment. URL: (visited 21/09/2015) Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,

17 Yusof, M. M., Kuljis, J., Papazafeiropoulou, A., and Stergioulas, L. K An evaluation framework for health information systems: human, organization and technology-fit factors (hot-fit). International journal of medical informatics, 77(6), Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul, Turkey,