Success criteria for electronic medical record implementations in low-resource settings: a systematic review

Journal of the American Medical Fritz F, et al. Informatics J Am Med Inform Assoc Association 2015;0:1 10. doi:10.1093/jamia/ocu038, Advance Access Review published March 13, 2015 Success criteria for electronic medical record implementations in low-resource settings: a systematic review RECEIVED 10 July 2014 REVISED 24 October 2014 ACCEPTED 26 November 2014 Fleur Fritz, Binyam Tilahun, Martin Dugas ABSTRACT... Objective Electronic medical record (EMR) systems have the potential of supporting clinical work by providing the right information at the right time to the right people and thus make efficient use of resources. This is especially important in low-resource settings where reliable data are also needed to support public health and local supporting organizations. In this systematic literature review, our objectives are to identify and collect literature about success criteria of EMR implementations in low-resource settings and to summarize them into recommendations. Materials and Methods Our search strategy relied on PubMed queries and manual bibliography reviews. Studies were included if EMR implementations in low-resource settings were described. The extracted success criteria and measurements were summarized into 7 categories: ethical, financial, functionality, organizational, political, technical, and training. Results We collected 381 success criteria with 229 measurements from 47 articles out of 223 articles. Most papers were evaluations or lessons learned from African countries, published from 1999 to 2013. Almost half of the EMR systems served a specific disease area like human immunodeficiency virus (HIV). The majority of criteria that were reported dealt with the functionality, followed by organizational issues, and technical infrastructures. Sufficient training and skilled personnel were mentioned in roughly 10%. Political, ethical, and financial considerations did not play a predominant role. More evaluations based on reliable frameworks are needed. Conclusions Highly reliable data handling methods, human resources and effective project management, as well as technical architecture and infrastructure are all key factors for successful EMR implementation.... Key words: Electronic medical record; hospital information system; implementation; low-resource setting; success criteria BACKGROUND Many low-income countries share the problem of not being able to provide sufficient health care services of high quality to their citizens, although they suffer from a great burden of disease. 1 About half of the World Health Organization (WHO) countries, most of them on the African continent, report to have less than one physician per 1000 individuals (eg, 0.008 in Tanzania, 0.18 in Kenya, 0.78 in South Africa) whereas the United States has 2.45 physicians per 1000 people, and in most European countries there are between 3 and 5 physicians. 2 Different approaches have been implemented to overcome this gap, for example, increasing the number of health workers and strengthening the health system through structural reforms. 3 One approach might also be to support the scarce medical staff by providing information technology (IT) for their everyday work, thus making information about their patients available and giving them the opportunity to document medical data in a structured format. This approach, in turn, produces valuable data to make sound decisions for the health care system. 4,5 Currently in most developing countries, the role of IT in the health care sector is mainly in so-called health management information systems, in which aggregated patient data are collected for reporting purposes. 6 These data can then be used to plan for needed competencies, to manage resources, and to produce indicators for public health. The importance of using IT in health care projects in the developing world was described by Fraser et al almost 10 years ago; however, many projects remain at the pilot installation level. 7 Furthermore, the weak health care infrastructure in lowresource countries results in a shortage of a qualified workforce and most health care institutions do not have dedicated IT staff for their electronic medical record (EMR) systems. 8 Any new project has to take this shortage into account and make efficient use of the available staff. Correspondence to Fleur Fritz, Institute of Medical Informatics, Albert-Schweitzer-Campus 1/A11, 48149 Münster, Germany; Fleur.Fritz@uni-muenster.de VC The Author 2015. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please email: journals.permissions@oup.com For affiliation see end of article. 1

Hence, to scale up systems, conserve valuable resources, and avoid any possible harm, health IT implementation projects need to be very carefully planned and based upon best practices. 9 A recent systematic review about the effect of EMR systems reconfirmed that evaluation studies are urgently needed and that barriers to implementation need to be addressed. 10 Therefore, evidence about success and failure criteria from previously or currently implemented health information systems are strongly required to make recommendations about the different factors that influence EMR implementation. Several research studies have reported on the most prominent criteria and have developed frameworks for future implementation projects. 11 13 In the specific context of developing countries, benefits and obstacles for hospital information system (HIS) use were reported at the end of the 1990s, 14 and many studies point out that HIS projects must ensure sustainability 15,16 and therefore need careful planning. In a review paper published about 10 years ago, Van der Meijden et al studied the determinants of the success of inpatient clinical information systems. 17 We, however, also want to examine the success criteria of an implementation project. Furthermore, our approach, while very similar, is to collect the attributes that determine success along with their measurements, if available, and then to categorize them. A recent nonsystematic review by Luna et al 18 focused on the sustainability of large-scale implementations at the country or regional level. They summarized their results about the current challenges in six broad categories and provided recommendations to policy makers and health care project leaders in developing countries. Another systematic review from Chaudoir et al 19 collected implementation success factors for health innovations in five different categories but they did not specify the setting. To our knowledge, no systematic review has summarized the experiences of institutions at the stages of deployment and early use of EMRs in low-resource settings. In this paper, we define the stages as being part of an implementation project, meaning the design, planning, rollout, and initial use of a system. Similar to the World Bank s definition of low-, middle-, and high-income countries, 20 we define low-resource settings in our context as those parts of the world in which the resources for health care (money, human resources, and technical infrastructure) are scarce. Sood et al summarized important differences between high-resource and low-resource settings with respect to EMR implementations. 8 The main characteristics are a weak health care infrastructure with inadequate funding and a lack of trained health care personnel; a rudimentary level of health care technology and inappropriate IT and power infrastructure; the existence of only basic training facilities and little health IT training within the medical curriculum; a political situation that heavily influences organizational dynamics, impeding efficient project management and often a large variety of different languages making it difficult to use English-based EMRs. There are many different kinds of information systems in health care and telehealth applications, in particular, are currently of great interest in many countries. Our focus, however, is on clinical systems and especially on those systems that focus on patient documentation primarily for clinical in- and outpatient services. For simplification, we refer to them as EMR throughout. Instead of discussing the successes and challenges during EMR implementations, we exclusively chose the term success criteria, which also includes its opposite. Our definition of success is based on the DeLone and McLean model of information systems success. 21 Their seven dimensions of information quality, system quality, service quality, use, intention to use, user satisfaction, and net benefits comprise a broad range of aspects and stakeholders. If we define the ultimate success to be a benefit for the user, all those dimensions will have a role to play. Objective The purpose of the present paper is to review EMR implementation projects in low-resource settings with respect to their reported success criteria. Our objectives in this review process are therefore to (1) identify relevant literature concerning implementation of EMRs in low-resource settings, to then (2) analyze EMR implementation projects and evaluation frameworks with respect to success/failure criteria, and to (3) summarize the results into recommendations for EMR implementation projects. METHODS Collecting literature We conducted a literature review in PubMed to identify success criteria and challenges of EMR implementation projects and the respective systems in low-resource settings. 22 In addition, the reference lists of relevant articles and additional articles by key authors were also reviewed. Full texts were obtained where needed. Two reviewers selected and reviewed all papers (FF, BT). The final search string for PubMed was: (Electronic health record [TIAB] OR Electronic medical record [TIAB] OR Electronic patient record [TIAB] OR health information system [TIAB] OR hospital information system [TIAB] OR care information system [TIAB] OR clinical information system [TIAB] OR computerized patient record [TIAB] OR digital medical record [TIAB] OR Computerized medical record [TIAB]) AND (developing countries [TIAB] OR low-resource setting [TIAB] OR resource-constrained setting [TIAB] OR resource poor setting [TIAB] OR resource limited setting [TIAB] OR Africa [TIAB] OR Asia [TIAB] OR South America [TIAB]). In a second search, the first half of the string was replaced with ( Medical Records Systems, Computerized [Mesh]) to explicitly verify the relevant MeSH term. We limited the results to those having an abstract and explicitly included the words Africa, Asia, and South America, because low-income countries are predominantly located on these continents. The main inclusion criterion was that the paper explicitly addressed EMR implementation challenges and experiences in a low-resource setting. Of the retrieved abstracts, we removed all papers not written in English. As our main interest focused on EMR implementations, we further removed those articles not related to the 2

Table 1: Seven categories for success factors of health IT implementation and additional details for further understanding Categories for success and failure criteria Ethical Financial Functionality Organizational Political Technical Training Additional details Includes regulatory and cultural issues Resources and funding System architecture and functions Focuses on (managerial) circumstances within the organization itself Includes legal health policies and country-wide circumstances Including infrastructure Including educational background and knowledge description of implementation projects or success factors of EMR usage in low-resource settings. During the full text review, we removed all papers that did not describe concrete success criteria of EMR and EMR-related systems implementations. A success criteria was selected as concrete if it was explicitly mentioned in the text, preferably in the results or discussion sections to avoid an interpretation bias. Furthermore, we added relevant papers retrieved through a review of the reference lists as well as hand-searched specific journals (eg, with a focus on developing countries). We did not distinguish between articles within conference proceedings or regular journals. A structured table of the inclusion and exclusion criteria can be found in S1. Literature analysis process As a basic framework for classifying the success criteria, we chose Brender et al s large Delphi study, 13 because it summarized the experiences of many HIS experts into six main factors for success and failure of health informatics applications. They are defined as functional, organizational, technical, managerial, cultural, and legal. Because of the above described differences between high- and low-resource settings, we decided to adapt some factors and specifically chose political, financial, trainingrelated, and ethical as our own categories. Consequently we did not explicitly use Brender et al s managerial, cultural, and legal categories, because we fitted the single factors they describe into our chosen category definitions. Our organizational category includes managerial issues and the cultural and legal issues were split up into an ethical and political category. Legal regulations with enclosed ethical problems are found in the ethical category. Those factors that deal mainly with health policies or are economically motivated were placed in the political category. We added the financial and training category because they are often referred to as being challenges in low-resource settings. That is also why our technical category includes infrastructure issues. Our resulting seven categories are listed in table 1 in alphabetical order. From the resulting relevant articles, the specific criteria as well as respective measurements were extracted if possible. This information concerned the country, the year of publication, the type of paper, the application area/system type, the disease domain, and the users. The type of paper was classified into evaluations, lessons learned, system concepts, implementation descriptions, success factors frameworks, reviews, and case studies either named explicitly in the paper or allocated by us. For the application area and type of system, we aimed at identifying the described system, whether it is an EMR, EHR, HIS, or any other of our terms in the search string. The disease domain means the domain for which the EMR was specifically designed or used. Summarizing We discussed each success criterion to come up with a congruent definition and wording of it. Wherever possible we tried to use the same wordings without changing the meaning. Within the seven main categories we defined subcategories based on the collected criteria in order to analyze the most reported ones. For all the collected information, we used descriptive statistical methods, such as frequency distributions, to analyze and summarize the results. RESULTS Search results after the collection and analysis process The PubMed search resulted in 190 articles as illustrated in Figure 1. After searching by hand in reference lists and selected journals, 33 additional articles were included. After reviewing titles and abstracts, we excluded 125 articles, as they did not fit our inclusion criteria; they did not report on EMR implementation projects. Furthermore, we excluded 15 articles that were not written in English (five) or for which the full text was not available (10). A total of 83 articles were then reviewed according to our defined criteria, of which 47 were included to extract success criteria and measurements, whereas the other 36 were excluded because of the above-mentioned reasons. Through analyzing the 47 publications, 381 success criteria and 229 corresponding measurements were extracted. Results after summarizing: general information about the articles We collected different information about the 47 papers included in our analysis. The complete list of publications with references and the reported information below can be found in S2. We categorized the publications into types of paper and found that a quarter of them were evaluation papers, followed by descriptions of a system concept and lessons learned as well as implementation. The last third were frameworks of 3

Figure 1: Literature review. Starting with 223 results coming from a PubMed research and 33 from other sources we identified 47 relevant papers, 176 articles had to be excluded. In our review we then extracted 381 success criteria and 229 corresponding measurements. success factors, review papers, and case studies. The numbers are depicted in Figure 2. In 64% of all retrieved publications, the EMR was implemented in a specific country, 36% reported in general about low-resource settings or sub-saharan Africa. From the specified countries, about three-quarters were from African countries. The country with the greatest number of publications was South Africa, as can be seen in Figure 3. The majority of articles were published in the last 10 years, with a recent peak in 2010 as depicted in Figure 4. The first publication was from 1999. From the evaluated or described implementation 80% were specified to be EMR and electronic health record systems, respectively; the other 20% were mentioned as general HISs. Almost half of those application systems served a specific disease area, especially for human immunodeficiency virus (HIV) care, as depicted in Figure 5. Users of the EMR were specified in only 10% of all publications, namely clinical staff, health professionals, nurses, and paramedics. There was almost no information about the number of users, dimensions of data captured, or how long the EMR had been used before the evaluation. Results after summarizing criteria of success The 381 success criteria are distributed into the following categories, sorted by frequency: 112, functionality (29%); 88, organizational (23.5%); 82, technical (21.5%); 37, training (10%); 24, political (6%); 21, ethical (5.5%); and 17, financial (4.5%). Almost half of the criteria (185) are mentioned more than once, 196 criteria are identified to be distinct. However, only 85 also have the same measures, which results in 296 unique criteria/measurement combinations. All single criteria can be found in S3. Figure 2: Types of papers. Most papers were of the type evaluations and system concept/lessons learned. The figure shows the absolute numbers as well as the percentages of all 47 papers. Within each category, single criteria are summarized into self-defined subgroups in order to obtain a better overview of the main issues that arise during the different phases of EMR implementation. Functionality: In this category, more than a third of all criteria focus on data handling in their different forms, eg, having a data dictionary, the quality of data, or data entry methods. Specifically named optional or extra features for the end users of the EMR are mentioned in 16% of the criteria. Around 10% reported on criteria of usability 4

Figure 3: Country distribution. The distribution in absolute numbers and percentages of countries (if specified in the 47 papers) in which the EMRs have been implemented and evaluated. Figure 4: Publication year. Number of publications per year from 1999 until 2013 of the 47 papers. and another 10% mentioned functionalities needed to aggregate and report data. Organizational: Within the organization, around 20% of the issues concern human resources, for example, having adequate skilled staff. Another 20% deal with issues of project management and the commitment to the EMR implementation project by the management. About 15% are criteria that stimulate or impede the use of the system and 15% are directly related to the attitude towards EMR. More than 10% of the criteria give evidence that the success of implementation depends on the local involvement of all stakeholders. Technical: Around 20% of the technical criteria reported on infrastructure, such as network and internet access as well as power supply. Approximately one-third are related to software architectural characteristics of the EMR. There are also concerns about security and privacy issues that need to be solved before an EMR can be successfully implemented (17% of the criteria). Almost 10% deal with standards including interface standards like HL7. Training: In this category half of the criteria are directly related to trainings, especially their availability. The other half is about existing knowledge and the background of 5

Figure 5: Disease area. Absolute numbers and percentages of the disease areas that the specific application system served, for example if the EMR was specifically used or designed for HIV care. This was only specified for almost half of the publications. the staff, predominantly the computer literacy of the users. Political: Half of the criteria addressing country-wide political issues and attitudes relate to characteristics like trust, attitude to change, and general political willingness. About a third relate to policies and the health care system infrastructure, for example the location of hospitals and the general information and communications technology setting. Ethical: Most of the criteria (48%) dealt with the sustainability of the implemented EMRs. Almost one-third were found to imply concerns of privacy and security. Financial: Almost half of the financial criteria were related to the availability of resources, most of them specifically mentioned human resources. General costs and the need for efficiency were mentioned in 37% of the criteria. For almost two-thirds (60%, 229 elements) of the success criteria, a specific measurement was mentioned that can also be found in S3; less than a third of them were used redundantly. In many cases, the measurement was merely the availability, absence, or level of fulfilment of the respective criteria. Others were more concrete like the user access rights with regard to system security, encryption methods for data privacy, or alternative power supply methods for electricity. The management commitment had different measures, for example allocation of resources or just a board membership. Cost effectiveness was simply measured by the cost benefit ratio; however, cost benefit assessment was measured by the vague indicator focus on local needs. Only very few measurement were based on standard instruments. DISCUSSION In this review, 47 out of 223 candidate papers published during the last 15 years are considered as relevant and represent a significant coverage of the field. In these papers, we identified 381 criteria that had a decisive impact on the outcome of an EMR implementation. Having collected such a large number of criteria gives a good reason for using these results. The results suggest that the main criteria for EMR success depend on the functionality of the implemented system, followed by the organizational structure and support for the project, as well as the availability of the technical infrastructure. Surprisingly, financing has not been found to be amongst the major criteria, although this seems to be an important factor as it is almost always initially mentioned, when talking about low-resource settings. The place of financing might be due to the fact that many of these projects were funded by donors, with the initial and on-going costs not being discussed. Lewis et al confirmed that almost half of the health IT projects in low- and middleincome countries are based on donor funding. 4 However, the financial aspect needs to be taken into account. Studies show that there is a return on investment in low-resource settings after 3 to 5 years. 23 The initial funding is therefore crucial to bridge the gap until the EMR generates a value to the health care institution. Otherwise, the argument could be made to use the budget for direct medical care. In this study, it seems that the success factors are similar to those found in non low-income countries. 24 Nevertheless, it should be recognized that differences exist in certain individual factors. The criteria of technical infrastructure, for example, often include a stable power supply and network, a problem which has been resolved in most high-resource settings. 6

Another example is the need for project support that might be achieved by strong support of leadership and efficient project managers. According to Sood et al, this management is often missing in low-resource settings because of a shortage of a qualified workforce and politics influencing organizational dynamics. 8 Limitations This work has several limitations. First, a publication bias is likely. In general, researchers from low-income countries have a low publication rate and evaluation studies, in particular, are substantially missing. There is also an imbalance in selection. In our search queries, we excluded papers not written in English. Consequently, there are fewer publications from French-speaking countries in Africa, even if our search string only resulted in fewer than five French publications. It might also be true that many of those foreign language studies did not get published at all in international journals. As we only included continent names in our search string and no single countries, the results of the country distribution from Figure 3 have to be used cautiously. In addition, we did not take into account the fact that some authors wrote several papers about their projects, reporting similar or the same criteria across those papers. We handled this in the same way as van der Meijden et al, who argued that they analyzed multiple evaluation studies from the same system separately when they described distinct aspects. 17 In the consensus process between the two authors, we discussed all criteria/measurements to ensure a shared understanding. For example the criteria sustainability or Involvement of local stakeholders sometimes appeared in the category organizational and sometimes political. Depending on the local or national focus we discussed to which category they belong, carefully reading the respective paper together. We sometimes also had to remind ourselves not to make our own interpretations of measurements but only to take into account what the authors were reporting. When required to improve comprehension, they have been renamed, conserving the meaning but making it easier for summarizing. Some of our choices might be debatable, but we tried to be as objective as possible. We also collected all described success criteria without weighting them. For example, OpenMRS is a largely implemented EMR system in many developing countries 25, and there are approximately eight published papers on its implementation. However, we took them all into account because they reported about different aspects of the system, about the implementation in different countries or settings, or about customizing it for various diseases or working conditions. Despite the above identified limitations, we believe that the number and type of publications analyzed in the review are representative for many low-resource settings. Other countries are encouraged to publish their implementation experiences. Gaps identified Concrete measures for the criteria were missing in a third of all criteria. When available, the formulation was often vague and not based on standardized instruments. Many publications were not real evaluations and thus did not measure success or failure of EMR implementation. Similarly to the WHO report of Tomasi in 2004, we had the impression that many papers were focused solely on opinions on the advantages and disadvantages of the use of IT, and lacked any evaluation of their concrete application to health care. 26 In our and their case, this might be attributed to the search strategy, which was not only focused on retrieving mere evaluation papers, but also any kind of EMR implementation projects. Important global information was usually missing, such as the numbers of users, how long the EMR had been used, the amount of data being processed, or where the funding came from. These points underline that detailed evaluation studies using robust and reliable methodologies are needed, especially in low-resource settings. 9 For example, guidelines on how to do and report good evaluations such as described in good evaluation practice in health informatics (GEP-HI) and statement on reporting of evaluation studies in health informatics (STARE-HI). 27,28 Related work In a recent review, Chaudoir et al 19 identified a framework of implementation success factors by means of outcomes that span the levels of structure, patient, organization, provider, and innovation. Most of the 62 measures collected from 125 articles, originated from the last three levels. Although our categories were different, we also identified many criteria in the organizational and few in the political/ethical category, being similar to Chaudoir et al s structural and patient levels. The authors suggest using defined measures and theoretical frameworks when constructing implementation evaluation studies to make research more reliable. This result is in line with our strong recommendation that reports on EMR implementations need to be grounded on sound evaluation criteria. Particularly for resource-constrained settings, another recent review from Oluoch et al focused on the clinical outcomes of decision support in EMRs. They report on implementation barriers. 10 The main criteria hindering implementations, according to Oluoch et al, were technical issues and training/ knowledge, power supply and computer skills, as well as the effective use of the system. Likewise, they showed that none of the 12 reviewed papers presented a strong evaluation design. Our review confirms these findings. The fact that issues of functionality matter a great deal was also stated by Heeks in a study assessing failure and success of health information systems. 11 He mentioned that there were differences between developing and industrialized countries, especially in HIS design, which might not match the user reality in low-resource settings and thus be more prone to failure. This assumption is also reflected by our findings from the functional criteria that the EMR needs to be adapted to the local needs. Heeks also suggested using his design-reality gap model to do preevaluations and risk management studies before starting the project and evaluate systems after implementation. 7

Luna et al reviewed challenges of health informatics implementation in developing countries. They focus on sustainability and recommended six areas for action : (1) resource and infrastructure limitations; (2) development of health IT agendas; (3) uncertainty in ethics and legal considerations; (4) a lack of interoperability standards; (5) skillfully trained workforce; and (6) regional integration efforts. 18 We agree that sustainability should be the basis of implementation strategies. However, when mapping these areas to our seven categories, we find that functional and organizational issues are missing. According to the results of our review, these should (also) be considered when conducting concrete implementations. Meaning and implications As discussed above, more evaluation studies with a strong methodology and comprehensive reporting should be conducted to create best practice recommendations for future EMR implementation projects. Importantly, many EMR implementation projects in lowresource settings were prototype systems or donor-based EMRs. This characteristic may be due to the special situation of having restricted resources or being in an early stage of adopting EMR system technology. In many situations, however, this poses the challenge of sustainability and scalability, for example, if the funding period does not foresee thorough training of the local staff, enabling them to manage first- and second-level support as well as further development. One possibility to overcome this issue was suggested by Braa et al 10 years ago: establishing networks that share experiences and possibly use the same software and training processes. 15 Consequently a future research question is whether that concept is still valid and, if so, why it is not applied more often. Furthermore, pilot projects need to foresee comprehensive handover strategies to the local staff. In low-resource settings, the installed EMR is often based on open-source systems, as was the case for approximately 45% of the analyzed papers in this study. Despite limitations in their functionality, special attention has to be paid to privacy, security, use of standards, and special user requirements, 29 which, as we know from this review, forms a large part of implementation success. An advantage of open-source systems is that the source code can be modified according to the needs, which potentially decreases the design-reality gap discussed above. 11 To reach this aim, it is recommended that the implementing organization gains the necessary capabilities and subsequently ensures the sustainability of the system. A positive example of how to adapt an open-source EMR was given by Fraser et al, who changed the OpenMRS to be used as a study management and documentation system. 30 This kind of multiuse brings advantages not only in low-resource settings, but might also be a success factor for other settings. The next step would be to integrate functionalities for clinical studies into EMR systems to promote the secondary use of clinical data and avoid redundant data entry. Furthermore, mhealth and telehealth are technologies growing in importance in these countries and should be seen in a broader context to complement comprehensive medical documentation. This paper, similar to previous reports, emphasizes that numerous factors may contribute to the success or failure of the deployment of an information system. 17 EMRs certainly have to be implemented in a holistic manner, taking into account numerous dimensions that affect them. According to the implementation projects we analyzed during our study, we recommend that in low-resource settings a special focus should be paid to functional, organizational, and infrastructural aspects. Regarding functionality, data handling issues and extra features are especially important. Aspects of usability and local adaptation are also crucial. In the organizational category, the key issue is to have staff with the right skills. Good project management and commitment by the decision makers are decisive, as is the involvement of the complete staff to promote a positive attitude towards the EMR. Regarding technical matters, two areas need special attention when planning an implementation project: the system architecture, including the use of standards and methods to ensure security and privacy, and infrastructure issues like a stable power supply and network. As a matter of fact, knowledgeable staff and the availability of training are basic requirements for successful health IT projects. Although sustainability plays a large role, financial issues, like resources and cost efficiency as well as general health care system infrastructure questions and policies, are not the primary challenges when looking at single implementations. Future research should promote robust evaluation of health care IT projects based on reliable and reproducible frameworks. A further review study could consequently analyze the impact of using the recommendations of evaluation reports during system implementation projects. Furthermore a list of available, evaluated EMR systems, designed especially for lowresource settings, is essential. Such a list should be multilingual and at least include functional descriptions, advantages/ disadvantages, training, and handover concepts for local support staff in the case of donor-based projects as well as all available characteristics that define the setting. This type of list would support decision makers and enable efficient and successful implementation projects. CONCLUSION Implementation projects of EMR applications in low-resource settings need to be carefully planned and ideally based on best practices to avoid wasting scarce resources. Implementers and policy and decision makers should not only focus on the finances but also, as demonstrated in the review, on functional, organizational, and technical aspects of the EMR. In particular, reliable data handling methods, human resources and effective project management, as well as technical architecture and infrastructure issues are key factors to make implementations successful. FUNDING This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. 8

COMPETING INTERESTS None. CONTRIBUTORS Fleur Fritz and Binyam Tilahun collected and read the literature, summarized success criteria, and did the data analysis. Fleur Fritz wrote the manuscript. Martin Dugas supervised the work and critically revised the manuscript. All authors read and approved the final manuscript. REFERENCES 1. WHO. The World Health Report 2010: Health Systems Financing: the Path to Universal Coverage. WHO Press, Geneva, 2010. 2. WHO Global Health Observatory: Density of physicians. http:// www.who.int/gho/health_workforce/physicians_density/en/. Accessed July 10, 2014. 3. Mills A, Rasheed F, Tollman S. Strengthening health systems. In: Jamison DT, Breman JG, Measham AR, et al, ed. Disease control priorities in developing countries. 2nd ed. Washington, DC: World Bank; 2006. Chapter 3. 4. Lewis T, Synowiec C, Lagomarsino G, et al. E-health in lowand middle-income countries: findings from the Center for Health Market Innovations. Bull World Health Organ 2012; 90(5):332 340. 5. Blaya JA, Fraser HS, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood) 2010;29(2):244 251. 6. Mutale W, Chintu N, Amoroso C, et al. Population Health Implementation and Training Africa Health Initiative Data Collaborative. Improving health information systems for decision making across five sub-saharan African countries: implementation strategies from the African health initiative. BMC Health Serv Res. 2013;13 Suppl 2:S9. 7. Fraser HS, Biondich P, Moodley D, et al. Implementing electronic medical record systems in developing countries. Inform Prim Care 2005;13(2):83 95. 8. Sood S, Nwabueze S, Mbarika V, et al. Electronic medical records: a review comparing the challenges in developed and developing countries. HICSS, 2008, 2014 47th Hawaii International Conference on System Sciences, 2014 47th Hawaii International Conference on System Sciences 2008, pp. 248, doi:10.1109/hicss.2008.141. 9. Rigby M, Ammenwerth E, Beuscart-Zephir MC, et al. Evidence based health informatics: 10 years of efforts to promote the principle. joint contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Med Inform. 2013;8(1):34 46. 10. Oluoch T, Santas X, Kwaro D, et al. The effect of electronic medical record-based clinical decision support on HIV care in resource-constrained settings: a systematic review. Int J Med Inform. 2012;81(10):e83 e92. 11. Heeks R. Health information systems: failure, success and improvisation. Int J Med Inform. 2006 Feb;75(2):125 137. 12. Brender J, Ammenwerth E, Nykänen P, et al. Factors influencing success and failure of health informatics systems a pilot Delphi study. Methods Inf Med. 2006;45( 1):125 136. 13. Gruber D, Cummings GG, LeBlanc L, et al. Factors influencing outcomes of clinical information systems implementation: a systematic review. Comput Inform Nurs. 2009;27(3): 151 63; quiz 164 165. 14. Azubuike MC, Ehiri JE. Health information systems in developing countries: benefits, problems, and prospects. J R Soc Promot Health 1999;119(3):180 184. 15. Braa J, Monteiro E, Sahay S. Networks of action: sustainable health information systems across developing countries. MIS Quarterly (Special Issue on Action Research in Information Systems) 2004;28(3):337 362. 16. Jacucci E, Shaw V, Braa J. Standardization of health information systems in South Africa: the challenge of local sustainability. Inform Techno Dev. 2006;12(3):225 239. 17. Van Der Meijden MJ, Tange HJ, Troost J, et al. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc. 2003; 10(3):235 243. 18. Luna D, Almerares A, Mayan JC, III, et al. Health informatics in developing countries: going beyond pilot practices to sustainable implementations: a review of the current challenges. Healthc Inform Res. 2014;20(1):3 10. 19. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013; 8:22. 20. World Bank: Country and Lending Groups. http://data.worldbank.org/about/country-and-lending-groups#low_income. Accessed July 10, 2014 21. DeLone W, McLean E. The DeLone and McLean model of information systems success: a ten-year update. JMIS 2003; 19(4):9 30. 22. PubMed. Bethesda (MD): National Library of Medicine (US). http://www.ncbi.nlm.nih.gov/pubmed. Accessed July 10, 2014. 23. Driessen J, Cioffi M, Alide N, et al. Modeling return on investment for an electronic medical record system in Lilongwe, Malawi. J Am Med Inform Assoc. 2013;20(4):743 748. 24. McGinn CA, Grenier S, Duplantie J, et al. Comparison of user groups perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Med. 2011;9:46. 25. Mohammed-Rajput NA, Smith DC, Mamlin B, et al. OpenMRS Collaborative Investigators. OpenMRS, a global medical records system collaborative: factors influencing successful implementation. AMIA Annu Symp Proc. 2011, pp. 960 968. 26. Tomasi E, Facchini LA, Maia MF. Health information technology in primary health care in developing countries: a literature review. Bull World Health Organ. 2004;82(11):867 874. 27. Rigby M, Ammenwerth E, Beuscart-Zephir MC, et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011;80(12):815 827. 9

28. Talmon J, Ammenwerth E, Brender J, et al. STARE-HI -statement on reporting of evaluation studies in health informatics. Int J Med Inform. 2009;78(1):1 9. 29. Flores Zuniga AE, Win KT, Susilo W. Functionalities of free and open electronic health record systems. Int J Technol Assess Health Care 2010;26(4):382 389. 30. Fraser HS, Thomas D, Tomaylla J, et al. Adaptation of a web-based, open source electronic medical record system platform to support a large study of tuberculosis epidemiology. BMC Med Inform Decis Mak. 2012;12: 125. AUTHOR AFFILIATION... Institute of Medical Informatics, University of Münster, Münster, Germany 10