How to Select a Clinical Information System

Transcription

1 How to Select a Clinical Information System Stefan Graeber Institute of Medical Biometrics, Epidemiology and Medical Informatics University of Saarland, Homburg, Germany Abstract Modern computer-based hospital information systems are mostly distributed with several information subsystems connected together by communication services. A single subsystem may be developed or purchased ('make" or "buy'). Before thepurchase of hard- and software it is necessary to check out whether the productfits the requirements concerning thefunctionality and especially the integration in the hospital information system. This selection process is difficult and tedious and requires thus a simple and flexible technique. An appropriate approach for the comparison of clinical information systems before implementation based on the SMARTER method is described here, and its practicability is demonstrated with an example. BACKGROUND A hospital information system is typically defined as that subsystem of a hospital which comprises all information processing actions together with the human and technical actors concerned in their corresponding information processing roles [1]. In order to map complexity and heterogeneity of such a comprehensively described hospital information system a distributed architecture has been introduced. This means that specific information systems for the organizational subunits of the hospital (clinical information systems, CIS) are combined together with powerful and versatile communication services [2]. According to the definition above, a CIS can be defined as the sum of hardware, software, people, procedures, and data which provides a spectrum of suitable functions. Examples for CIS are a laboratory information system or a radiology information system A much-used model of CIS development and operation describes the progress of a CIS through time as an endlessly repeating life-cycle [3]. In each stage the hospital's information management must be able to define quality criteria and to evaluate the state of a CIS (life-cycle of evaluation). We focus here on the time before implementation, i.e. once the strategic plan is understood and there is some kind of consensus as to its nature and measure, the evaluation provides information which enables the management to (among other things) [3] * Satisfy itself that the chosen solution is feasible, e.g. that the hospital has the resources and skills, * Identify the obstacles which might make the chosen course of action difficult or even impossible to achieve, * Choose between alternative solutions and designs, e.g. the choice between a new development ("make") or selection, purchase, and customizing of conmercially available CIS products ("buy"). In maldng the decision between "make" or "buy", the preferred strategy primarily depends on the resources ofthe hospital. The present paper discusses the situation where the decision to "buy" has been made: one has to find appropriate products and to compare them before investing. A useful comparison requires a formal evaluation, which concentrates on the scope and quality of functions, the performance, and the costs of the CIS. Many evaluation techniques have been suggested to enable organizations to make the correct choices, e.g. the Analytic Hierarchy Process [4] or outranking approaches [5]. We describe here a method for decision analysis for the selection of CIS based on the Simple Multi-Attribute Rating Technique Exploiting Ranks (SMARTER) [6] and our experiences gained with its application in the setting of a large university hospital. METHODOLOGY SMARTER is a method to analyze decisions involving multiple objectives. Attributes are used to measure performance in relation to an objective. The final score assigned to an alternative will be referred as the value of an option. Goodwin and Wright [7, p.18] propose eight main stages in performing a decision analysis. Identify the decision maker(s) As part of the tactical management a project group (committee) with members of several hospital fields (physicians, nurses, technical assistants, medical informatics experts, administrational staff, and possibly external consultants) has to be installed. Hwang and Lin [8, p.271] define four sequential phases of decision making with different tasks for the group. The detailed explanation of content, scope, require /01/S AMIA, Inc. 219

2 Value ofris..~~~~~~~~~~~~~~~~~~~~~ Benefits Costs Quality of functions - Admission - Examination - Management of findings - Secretarial functions - Archiving - Special functions Performance Investment Maintenance Prerequisites Purchase Contracts Customizing Consulting Support Interfaces Training -.. Data security Clinical processes Organizational impacts Figure 1: Part of value tree for a radiology information system (RIS) ments, and analytical methods of the CIS project In order to identify the attributes usually the method according to the objectives defined by the strategical of brainstorming by the project group together with a management forms the preparatory phase. top-down approach is applied. Based on the experience of the members a value tree is constructed, After the screening phase (selection of alternatives) and the evaluating phase (comparison of alternatives), breaking down the general attributes into more specific attributes and considering that the attributes in the decision phase the project group makes a recommendation to the board of directors. must be mutually preference independent (figure 1). In the screening and evaluation phase the project A data dictionary contains an extensive description group forms a panel of judges. Judges are humans for each attribute. Functions are arranged in logical with particular expertise who provide their informed clusters or groups (for an example see table 2). opinions about behavior they observe from some Attributes can be represented by qualitative variables, record of that behavior [9, p. 130]. except costs and some times and numbers measuring In this context the time for the evaluation is limited performance. For these latter linear value functions usually to three months. The medical informatics staff have to be constructed, which map the quantitative trains the members of the project group in defining scale into a qualitative value scale. the basics and handling the methods and tools. The set of attributes is checked for completeness Identify the alternative courses of action The first task in the screening phase is to fix some key criteria (e.g. the existence of some functions or interfaces or cost limits) with defined cut-off points. After a market survey all products which fall below a cutoff in any criteria, are eliminated. We get a short list (not more than six products) to be evaluated in detail. Identify the relevant attributes The relevant properties of the alternatives have to be described in the form of attributes. Each attribute should be able to differentiate between the options. An attribute, cost for example, may be important in its own right, but if all options cost the same then it is redundant in the evaluation of the options. It is not easy to decide the right level of granularity. As the items become more global, agreement between raters is difficult to achieve. As they become more atomic, the evaluation may be mechanical or trivial [9]. (nothing important omitted), operationality (the granularity is specific enough to compare the options), decomposability (the effect of an option on one attribute can be judged independently of its effect on other attributes), and absence of redundancy (any attribute which does not differentiate between options is eliminated) [7, p.21]. Measurement of attributes The costs may be determined directly or estimated using similar projects. For other quantitative attributes the measurements are transformed using value functions. The qualitative variables on the lowest level in the value tree are rated directly on an ordinal scale from 0 to 3 (table 1, score I). While data processing the scores are mapped automatically to a 0 to 100 scale (table 1, score II). In our opinion, for this purpose four discrete options suffice. More options would lead to a deceptive accuracy. Due to the central ten- 220

3 dency problem [9, p.146] we do not provide a neutral response, i.e. the number of scores is chosen to be even. This forces the rater to state a clear position. Table 1: Rating score Quality Score I Score U Not sufficient 0 0 Sufficient 1 33 Good 2 67 Excellent The judge appraises the attribute and assigns a fitting score. Each member of the con ittee evaluates its own working field (e.g. a clinician the functions to support physicians). At least three people should rate each attribute. They collect information about the alternatives by * presentations by the sellers, * visits to reference installations (site visits), * tests at reference installations. Each judge must assign a score, unless he does not have the professional knowledge to appraise the attribute, i.e. all members of the project group should be involved in the evaluation process. If there are severe differences between the judges the medical informatics staff decides. The results of scoring are entered in the intermediate criteria matrix (table 2). Determine a weight for each attribute The weight of an attribute reflects its relative importance for the decision makers. In SMARTER socalled Rank Order Centroid (ROC) weights are used to rank the attributes on the same level in the value tree. These weights only depend on the number of attributes considered [6]. The committee must rank the attributes and determine the weights. Calculation of scores For each node in the value tree scores are calculated as weighted means of the single scores (additive model, table 2) and the results from all judges are averaged together nodewise as proposed in [8]. Thus it is simple to derive a final criteria matrix for the benefits as shown in table 3. All calculations are performed with Microsoft EXCEL. Provisional decision In this stage a cost-benefit analysis should be performed. A scatter plot with the average weighted score for the benefits and costs is plotted (figure 2). The cost scale is inverted, i.e. the lower costs are to the right. The dashed line shows the "efficient frontier" [7, p.33]. The choice between the options on the frontier depends on the relative weight which is attached to benefits and costs. This presentation is the base for a provisional decision. 07 r-. %fvoo-ylil. ^ I I I I _,I I l I - I Costs [Mio DM] Figure 2: Plot of benefits against costs for the selection of a radiology infortion system (RIS) Sensitivity analysis A sensitivity analysis is used to examine how robust the choice of an alternative is to changes in one or more key input variables. Sensitivity analyses are performed as described in [6]. RESULTS At the University Hospital of Saarland this approach has been developed since 1996 in preparation of the introduction of an operating room IS, a laboratory IS, a pathology IS, a radiology IS, and a clinical workstation system [10, 12]. As an example the project to introduce a radiology information system (RIS) with a short list of five products is presented here. The most important logical clusters are (see also figure 1): * Admission (scheduling, access to patient management, working lists) * Examination (recording of data and pictures) * Management of findings (see table 2) * Archiving (registration of files, documentation of loans, access to PACS) * Customizing (features to adapt the software according to user requirements) * Interfaces (user interface, HL7, standards) * Expected impact on clinical processes (reduction of waiting times, speed of access to information). The project group for the RIS had 15 members. Each item could be assessed by at least three judges. Table 2 shows a part of the intermediate criteria matrix and table 3 the final criteria matrix for the benefits. Product C had the highest score but with relatively small difference to A and B, while D and E were inadequate. Interpreting the cost-benefit plot (figure 2) we can see that C is dominating over A and thus the choice depends on the amount to be invested only. In this case option B was purchased, because the committee judged the benefit difference between B and C as less relevant than the cost difference. 1% D 221

4 Table 2: Part of intermediate criteria matrix (RIS evaluation, one judge) Attribute of function group ROC Score Score Score Score Score "Management of findings" Weight Product A Product B Product C Product D Product E Working lists Patient identification via barcode View of previous findings Use oftextblocks Voice-messaging Speech input and decoding Weighted mean Table 3: Part of final criteria matrix for the benefits (RIS evaluation) ROC Score Score Score Score Score Attribute (n=16) Weight Product A Product B Product C Product D Product E Admission Exaimination Management of findings Prerequisites Customizing Interfaces Aggregate benefits A post-implementation evaluation of the RIS is still running and will be published separately. Early results show that * the overall functionality is sufficient, * the nurses and technical assistants are satisfied and can handle the system without problems, * the physicians have some difficulties to become familiar with the system, possibly due to inadequate training, * the customizing of the software took longer and was more expensive than expected. DISCUSSION In applying SMARTER one has to make the following assumptions (axioms [7, p.37]): * The decision does not involve elements of risk or uncertainty. This assumption is satisfied sufficiently in the purchase situation cited above. * The decision maker is able to decide which of two options he prefers (decidability). * The value measure is transitive and satisfies the triangle inequality. * There are finite upper and lower bounds for values. * The attributes are mutually preference independent. This is a prerequisite for application of the additive model. SMARTER convinces by its relative simplicity of both the responses required of the decision makers and the manner in which these responses are analyzed. In contrast, outranking methods for example (like PROMCALC&GAIA [5]) require more skill of the judges and more support by specific computer programs. Nevertheless, SMARTER provides an approximation of the real decision situation only. In practice, the approach has been found extremely robust [7, p.18]. The main purpose of the decision analysis is insight not numerical treatmnent. Definition of attributes, elicitation and use of values contribute to emergence of insights in important ways [6] (see below). The validation of the method is difficult. The content validity may only be assumed if standardized catalogues for the scope of functions of CIS are used. Such reference catalogues for example have been published by the German association for medical informatics (GMDS) [13]. Dedicated studies to estimate criterion-related and construct validity [9] have not been performed up to now. In our experience the outlined approach provides usefiul conclusions when selecting a CIS: * The axioms and principles are reasonable and the whole procedure of introduction of a CIS becomes transparent and controllable. The decision may be better defended. The results (especially value tree and data dictionary) may be used as a reference model for the functional description of the CIS, e.g. for the post-implementation evaluation [10]. 222

5 * Before the introduction of this approach decisions about the purchase of a CIS were made intuitively by a few people. The new method coerces the parties concerned to define a clear structure of the requirements, to investigate the possible products accurately, and to make the final decision with care and specific understanding for the balance between wishes and reality. This comprehension increases the motivation for the implementation of the choice too. * Due to the number of items the power of the rating procedure is relatively high, i.e. the differences between the products become apparent. On the other hand there are some disadvantages: * The effort ofjudging involved is considerable. * The complexity of the systems evaluated may obliterate the differences between the products. * The judges can be prejudiced in favor of a product and the results are related to the investigator's professional training and background. * Although the evaluation process should be performed within three months, the information becomes rapidly out of date. Products are changed frequently and the results tend to become invalid. Furthermore, it is difficult to get the relevant information for the evaluation of such complex systems: the sellers do not tell the whole truth or they keep something secret, demonstrations of software are inadequate (and therefore have to be steered strictly), and information on the further development of the product is not always disclosed. Therefore a stringent controlling of the whole evaluation process is necessary, if possible by an "external" expert team. In our setting, members of the medical informatics staff not involved in the project group do this work. In our opinion the evaluation scheme described is feasible. We use it as a pragmatic tool to handle the complexity of a CIS and to make evidence-based and consensus-based decisions. REFERENCES [1] Winter A, Brigl B, Buchauer A et al. Purpose and Structure of Strategic Plans for Information Management in Hospitals. Stud Health Technol Inform. 2000;77: [2] Graeber S. Communication Services for a Distributed Hospital Information System. Methods Inf Med. 1996;35(3): [3] Farbey B, Land F, Targett D. How to Assess your IT Investment. Oxford: Butterworth- Heinemann; [4] Saaty TL. Multicriteria Decision Making - The Analytic Hierarchy Process. Pittsburgh: RWS Publications; [5] Brans JP, Mareschal B. PROMCALC&GAIA: A new decision support system for multicriteria decision aid. Decision Support Systems 1994;12: [6] Edwards W, Barron RH. SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurenment. Organizational Behaviour and Human Decision Processes 1994;60: [7] Goodwin P, Wright G. Decision Analysis for Management Judgment. 2nd ed. Chichester: Wiley; [8] Hwang CL, Lin MJ. Group Decision Making under Multiple Criteria. Berlin: Springer; [9] Friedman CP, Wyatt JC. Evaluation Methods in Medical Informatics. New York: Springer; [10] Hofinann K, Graeber S. Evaluation of Clinical Workstations. In: Brender J, Christensen JP, Scherrer JR, McNair P, editors. Medical Informatics Europe '96. Amsterdam: IOS Press; p [11] Graeber S. Application of Clinical Workstations: Functionality and Usability. Chin Perform Qual Health Care. 1997;5(2):71-5. [12] Ikas/index.html (in german) 223