Answering Clinical Questions Improves Patient Safety and Saves Money

Transcription

1 Answering Clinical Questions Improves Patient Safety and Saves Money The Hospital Clinical Knowledge Systems and Health Outcomes Study Peter A.L. Bonis, M.D. Executive summary: Improved care results in cost savings Several tools are currently being deployed in provider environments to improve quality and efficiency of care. Chief among these are electronic medical records (EMR), computerized physician order entry (CPOE), E-prescribing, and medication bar-coding systems. UpToDate Hospitals with access to UpToDate performed significantly better on patient safety, complications, and length of stay resulting in a strong return on investment. However, it is imperative that quality improvement initiatives also include a clinical knowledge system. Physicians generate one question for every two or three patient encounters and two-thirds of those questions go unanswered. If physicians had access to the appropriate clinical information system, and could answer clinical questions as they arise, some 40 percent of management decisions would be changed, improving patient care. Regularly updated electronic clinical knowledge systems have decreased barriers to answering these questions. However, there has previously been little evidence as to whether they have an impact on health outcomes. The Hospital Clinical Knowledge Systems and Health Outcomes Study was designed to evaluate whether a leading clinical knowledge system (UpToDate ) was associated with improvement in patient safety and efficiency in hospitalized patients, and to estimate the financial impact associated with its implementation. The study demonstrated a strong association between hospital quality and efficiency and use of UpToDate. Hospitals with access to UpToDate (n=424) performed significantly better on risk adjusted measures of patient safety (p=0.0163) and complications (p=0.0012) and had significantly shorter length of stay (by an average days per discharge, 95% confidence interval to days, p<0.0001) compared with hospitals without access. These associations persisted after adjusting for hospital characteristics known to be associated with the outcomes including geographic location, teaching status, and discharge volume. 1.

2 The study also confirmed a strong correlation between how much UpToDate was used at each hospital reflecting a dose-response effect. For example, a doubling of usage from 26,000 to 52,000 topic reviews annually in a hospital with 20,000 annual admissions could save an additional 520 hospital days each year. Metrics used in the 100 Top Hospitals differentiate hospitals on a variety of performance dimensions such as quality and efficiency. They have been used in many previous studies evaluating healthcare quality UpToDate provided Solucient with a list of hospitals that had online access to UpToDate from 2000 through 2005 as well as usage data detailing how much UpToDate was used at each hospital. Solucient compared outcomes in Medicare recipients who were treated at hospitals with online access to UpToDate to those in acute care hospitals included in the Solucient 100 Top Hospitals database. For more specific information on the chart above, see page 3. Research study supports the value of UpToDate Background Clinical knowledge systems help providers answer clinical questions. A large body of evidence has demonstrated that most clinical questions are not pursued even though the answers to the questions would have a favorable impact on management decisions Physicians generate about one question for every two to three patient encounters and two-thirds of those questions go unanswered. If answered, approximately 40 percent of management decisions would be changed. 11 The ability to answer clinical questions quickly has the potential to improve patient safety and hospital efficiency. The outcomes evaluated were riskadjusted for mortality, complications, the Agency of Healthcare Research and Quality Patient Safety Indicators, and hospital length of stay. These outcomes were chosen because they reflected the types of benefits that might be expected from a knowledge system that has been designed to answer clinical questions. Details on the study methods and statistical analyses are provided in Appendix 2. The Hospital Clinical Knowledge Systems and Health Outcomes Study was designed to assess whether the use of a leading clinical knowledge system, UpToDate, (see Appendix 1) was associated with quality of patient care and the economic implications of decreased length of stay and complications, and increased patient safety. 26 Methods Solucient (now a subsidiary of Thomson Healthcare) is a well recognized leader in measuring hospital performance. Each year it reports on the top performing hospitals in the United States (the Solucient Top 100 report, now the Thomson Top 100 report) based upon analysis of a database that contains information regarding inpatient Medicare admissions from 3,091 short-term general, non-federal hospitals in the United States (i.e. approximately 75 percent of the 4,200 acute, non-federal hospitals in the United States). 2.

3 Results Hospitals with access to UpToDate (n=424) performed significantly better than other hospitals in the Solucient database (n=3,091) on risk-adjusted measures of patient safety (p=0.0163) and complications (p=0.0012), and had significantly shorter length of stay (by an average days per discharge, 95% confidence interval to days, p<0.0001). These associations persisted after adjusting for hospital characteristics known to be associated with the primary outcomes including geographic location, teaching status, and discharge volume. The study demonstrated a strong linear association between hospital efficiency and safety, suggesting that UpToDate represents an important component of hospital quality. UpToDate allows providers to answer questions quickly, leading to changes in decision-making that can improve management and efficiency. 11,17,36 In addition, UpToDate is an important factor in medical knowledge acquisition among medical residents. 37 These effects, along with time savings and early adoption of efficient diagnostic and management strategies, may explain the observed benefits on patient safety measures and length of stay. Financial implications of improved efficiency and patient safety In addition, all of these associations correlated significantly with how much UpToDate was used (reflected in hits per week [HPW], the average number of topic reviews viewed each week) supporting a dose-response relationship. The graphs depict the relationship between hospital discharge volume, the amount that UpToDate was used, and the observed benefits on complications and length of stay. The cut points on HPW reflect the quartiles of the data. The improved efficiency and patient safety benefits associated with use of UpToDate would be expected to have substantial economic benefits which were estimated by an analysis performed by UpToDate. The frequency and costs associated with adverse events are somewhat more difficult to estimate. The Solucient data, which are based on medical claims, likely underestimate the frequency of adverse events since claims-based data are known to be insensitive to adverse events. 38 The cost of an adverse event was estimated to be $30,936 based on a study of preventable medical injuries in 28 hospitals in the United States adjusted to 2006 dollars. 39 Notably, substantial additional financial benefits would be expected for payers and from a reduction in costs to society, factors that were not included in the models. 3.

4 The financial impact of UpToDate is directly associated with the amount that UpToDate is used at each hospital. Use grows spontaneously year after year (both in the number of users and the amount of use for each user) at hospitals that subscribe. Thus, the economic benefits described above would grow proportionally to the rate of adoption and the amount that UpToDate is used. Efforts to stimulate use of UpToDate could further increase the benefits. As the data suggest, these systems can improve care and hospital efficiency producing a large return on investment comparable to if not exceeding the expected return on investment for implementation of CPOE systems. Thus, they should be considered complementary and an essential component of efforts to improve hospital efficiency and safety. Relationship between use of UpToDate, complications avoided and hospital days saved each year for a hospital with 20,000 annual discharges. The graph on the left shows that the number of complications that can be prevented increases with greater use of UpToDate. The complications evaluated considered the underlying severity of illness. They included conditions that should not occur and thus represent substandard care, and complications that occurred with a frequency more than expected. The complication methodology has been used in many studies evaluating patient safety. The graph on the right shows that the expected number of hospital days that can be saved increases with greater use of UpToDate. The estimates were based upon the regression model and considered case mix and hospital characteristics known to be associated with hospital length of stay. (See Appendix 2). Whether the observed associations with hospital quality and efficiency are applicable to all electronic knowledge resources is unclear. It is likely that the benefits from an electronic clinical knowledge resource would be related to the degree to which it is adopted and used. Spontaneous usage levels for UpToDate increase year after year as a result of increased use among providers and an increasing number of users. The return on investment compares favorably with other technological innovations aimed at improving the quality, safety, and efficiency of care. Relatively few studies have directly evaluated the return on investment for information technology initiatives. 40,41 A landmark study involving six Massachusetts community hospitals estimated that implementation of a CPOE system could save each hospital $2.7 million annually. 42 The one-time average total cost of a CPOE system was estimated to be $2.1 million with an annual incremental operating cost of $435,000. Most of the savings were anticipated to result from avoidance of preventable adverse drug events, which were estimated to occur in 10 percent of hospitalized patients. The economic case for CPOE systems is strong based upon these data but the magnitude of expected benefits is less than those associated with the implementation of clinical knowledge systems. The chart depicts the median topics viewed each week at 114 hospitals with 18,000 to 25,000 admissions each year. Usage shown is for the year and is stratified according to how many continuous years each hospital had subscribed. Use increased significantly (p<.0001). Clinical knowledge systems can also be implemented far more easily than decision support tools embedded within CPOE or EMR systems. The challenges involved in successful implementation of EMR and CPOE systems with acceptable decision support are well-recognized. 43,44 By contrast, implementation of clinical knowledge systems can be accomplished easily and with enthusiastic acceptance by providers. 4.

5 The acceptance of UpToDate among providers has translated into widespread adoption. Over 88% of academic medical centers in the United States subscribe to UpToDate and nearly 320,000 healthcare providers use it worldwide. About 98 percent of hospitals that subscribe to UpToDate continue to renew their subscription year after year. UpToDate users at these hospitals depend on it for their practices. Hospitals that subscribe to UpToDate frequently report that providing access is important for recruiting and retaining their physicians. Studies of users in community hospitals have demonstrated that use is not confined to doctors. UpToDate is also used extensively by nurses, nurse practitioners, physician assistants, and pharmacists. This permits UpToDate to become the standard of care, thereby helping to ensure consistent care that is well-integrated across all caregivers. This degree of use means that UpToDate impacts more than 90 million patient visits a year in more than 140 countries. Use has an impact on learning The widespread adoption and use of UpToDate has had an impact on learning among providers and trainees. A study performed by the Mayo Clinic, for example, examined the impact of learning habits (use of UpToDate and conference attendance) on medical knowledge acquisition as measured by the Internal Medicine In-training Examination (IM-ITE). 37 The IM-ITE assesses medical knowledge acquisition during residency; scores correlate with subsequent performance on the American Board of Internal Medicine Certification Examination. Use of UpToDate was an independent predictor of performance. Remarkably, use for 20 minutes a day was associated with a comparable increase in IM-ITE scores as an entire year of residency. The most frequently used methods to retrieve clinical information were Medline (73%) and UpToDate (70%) in a study of residents from the University of Colorado Health Sciences Center. From Schilling, L. Acad Med 2005; 80:51. Methods to increase use Increased benefits from clinical knowledge support systems can be realized by increasing their usage. For example, a doubling of UpToDate usage from 500 to 1000 topic reviews per week in a hospital with 20,000 annual discharges could result in more than 520 hospital days saved each year (and more than $800,000 in annual savings). Increasing use can be accomplished by: Increasing awareness and training Reducing the click distance to the resource by integrating it into the EMR system Rewarding providers for use by granting continuing medical education credits Conclusions and recommendations Many factors are involved in hospital quality and efficiency. The value of clinical knowledge systems such as UpToDate as part of efforts to improve care has not been fully recognized. The Hospital Clinical Knowledge Systems and Health Outcomes Study demonstrated that use of an electronic clinical knowledge system was associated with hospital quality and efficiency. Such systems can improve care and efficiency and are well-accepted by providers. The benefits on patient safety and efficiency also translate into a large return on investment, and require minimal resources to be implemented. That return on investment can be continually improved by efforts to increase usage. The march toward broad implementation of electronic medical record, CPOE, E-prescribing, and medication bar-coding systems is continuing. The costs and complexity involved in implementing such systems have overshadowed the comparatively easily acquired benefits of implementing clinical knowledge support systems. Such systems can be implemented rapidly leading to almost immediate improvements in patient care. In addition, these systems are important for education of trainees and continuing medical education of providers. 5.

6 Appendix 1: Description of UpToDate UpToDate is an evidence-based, peer reviewed information resource available via the Web, desktop, and PDA. UpToDate is designed to answer clinical questions quickly and provide synthesized recommendations for treatment. Over 90 million patient-related problems are researched each year with UpToDate. The UpToDate community includes our faculty of more than 3,800 leading physicians, peer reviewers, and editors and nearly 320,000 users. Our faculty writes topic reviews that include a synthesis of the literature, the latest evidence, and specific recommendations for patient care. Our users provide feedback to the editorial group. This community s combined efforts result in the most trusted, unbiased medical information available. UpToDate is available through institutional or individual subscriptions. Providers at hospitals that subscribe to UpToDate can access it through an Internet connection at any terminal within the facility. UpToDate content UpToDate includes more than 7,400 clinical topics, each an average of 10 pages of text, figures, references, and abstracts, as well as drug and drug interactions programs, guidelines, patient handouts, and clinical calculators. Access to the content of UpToDate is through a search screen with a search box. UpToDate uses a proprietary search engine. The search engine is not static but undergoing continual adjustments to provide the best search results given the content offering. The search engine can handle multiple concepts from a natural language question like: Treatment of hypertension in a pregnancy. The search concepts here are: treatment, hypertension, and pregnancy. During the search, the entire database gets reorganized in order of relevancy. Authors All topics in UpToDate are written by the listed authors in conjunction with a deputy editor. Authors are identified as experts by the Editors-in-Chief, our editorial staff, and participating societies. Exceptions are guidelines from major societies, which are added to UpToDate in their original form. All material is originally prepared by the contributing author(s) whose name(s) and affiliation(s) appear in the upper left corner of each topic. This material is reviewed extensively by our physician editors and peer reviewers for accuracy and completeness of the literature search, and for consistency with all aspects of the editorial policy. Physician editors suggest changes to ensure that the topics summarize the relevant evidence, and that the recommendations are consistent with the evidence, with our understanding of patients values and preferences, and with UpToDate s editorial policy. Some of the content may be taken from other topics in UpToDate. In such cases, the text is hyperlinked to the topic from which it originated. Evidence UpToDate follows a hierarchy of evidence consistent with most evidence-based resources. At the top of the hierarchy are randomized trials of high methodological quality, followed by randomized trials with methodological limitations, observational studies, and unsystematic clinical observations. Inferences are stronger when the evidence is summarized in systematic reviews of the literature that present all relevant data. Each topic has an author who is an expert in the area discussed, and at least two separate physician reviewers. This group works together to perform a comprehensive review of the literature and carefully select studies for presentation based upon the quality of the study, the hierarchy of evidence discussed above, and clinical relevance. 6.

7 When current, high-quality systematic reviews are available, UpToDate topics and recommendations rely heavily on these reviews. When such reviews are unavailable, UpToDate summarizes the key studies bearing on the clinical issues at hand. Systematic reviews and the design of primary studies (randomized trial, observational studies) are often identified explicitly in the text with the relevant data. However, in cases where either the type of study or the data are not stated explicitly, users can click on the reference and bring up the Medline abstract to obtain this information. Evidence is derived from a number of resources, including but not limited to: Hand-searching of over 400 peer reviewed journals Electronic searching of databases including MEDLINE, The Cochrane Database, Clinical Evidence, and ACP Journal Club Guidelines that adhere to principles of evidence evaluation described above Published information regarding clinical trials such as reports from the Food and Drug Administration, as well as other sources of information produced by federal agencies such as the Centers for Disease Control and Prevention and the National Institutes of Health Proceedings of major national meetings The clinical experience and observations of our authors, editors, and peer reviewers Recommendations UpToDate strives to provide a Summary and Recommendations section (see figure below) where appropriate. Structured questions UpToDate s process of arriving at recommendations involves constructing a structured clinical question. That structure includes carefully defining the patient population of interest, the alternative management strategies, and the outcomes of importance to patients. Values and preferences A fundamental principle of evidencebased medicine, as described by Dr. Gordon Guyatt from McMaster University, is that Evidence alone is never sufficient to make a clinical decision. Decision makers must always trade the benefits and risks, inconvenience, and costs associated with alternative management strategies, and in doing so consider the patient s values. 45 This principle has led some evidencebased resources to avoid making specific recommendations for patient care, since the recommendation needs to account for all of the factors cited. UpToDate has taken a different approach. It is the policy of UpToDate to make specific recommendations for patient care whenever possible. Recommendations in UpToDate are based upon a synthesis of evidence including that from clinical trials and clinical experience; whenever possible, the evidentiary basis for recommendations is stated explicitly. When there is no published systematic evidence available (eg, prednisone dosing regimen in pulmonary sarcoidosis), recommendations are based upon the unsystematic clinical observations of our experts and reviewers, and on pathophysiologic rationale. 7.

8 UpToDate recommendations identify situations in which different decisions might be appropriate for patients with different values and preferences. Furthermore, UpToDate recognizes that recommendations will not apply to every patient, and counts on clinicians to evaluate the recommendations in light of the individual circumstances of their patient. Nevertheless, UpToDate feels that giving clinicians access to recommendations based on a sophisticated understanding of the clinical issues, the best evidence, and a consideration of patient values and preferences, allows them to make informed decisions with and for their patients. Grading UpToDate began grading recommendations for treatment and screening in This is a continuing process and not all such recommendations have yet been graded. Graded recommendations appear in the Summary and Recommendations sections at the end of topics. UpToDate uses the UTD-GRADE format, a modification of the GRADE system. 46,47 Grades have two components, a number (1 or 2) reflecting the strength of the recommendation and a letter (A, B, or C) reflecting the quality of the evidence supporting that recommendation. A Grade 1 recommendation is a strong recommendation to do (or not do) something, where the benefits clearly outweigh the risks (or vice versa) for most, if not all patients. A Grade 2 recommendation is a weaker recommendation, where the risks and benefits are more closely balanced or are more uncertain. The majority of recommendations will be Grade 2 recommendations. UpToDate uses wording that reflects the strength of the recommendation: strong (Grade 1) recommendations are recommended and weak (Grade 2) recommendations are suggested. Grade A evidence means high-quality evidence that comes from consistent results from well-performed randomized controlled trials, or overwhelming evidence of some other sort (such as well-executed observational studies with very strong effects). Grade B evidence means moderate-quality evidence from randomized trials that suffer from serious flaws in conduct, inconsistency, indirectness, imprecise estimates, reporting bias, or some combination of these limitations, or from other study designs with special strength. Grade C evidence means low-quality evidence from observational evidence, or from controlled trials with several very serious limitations. 8.

9 Moving from evidence to recommendations The following table presents the criteria that UpToDate authors and editors consider when weighing the advantages and disadvantages of treatments to decide on a recommendation, and grade the strength of that recommendation. Each topic has a date indicating the most recent time the topic has been reviewed and/or modified. On average, approximately 40 percent of the topics are updated during each four-month cycle. Peer review The Deputy Editor for a specialty, as well as the Editor-in-Chief and/ or Section Editors assigned to a topic, review the entire UpToDate content, including all new topics, updates, and recommendations. In addition, each UpToDate specialty has assembled a group of peer reviewers, often in conjunction with a sponsoring specialty society, that is responsible for reviewing selected topics in each specialty. Finally, any comments from users of UpToDate are formally addressed with changes made as necessary. Updating UpToDate performs a continuous comprehensive review of resources previously outlined to keep the program updated. All of the topics in UpToDate are revised whenever important new information is published, not by any specific time schedule. Updates are integrated carefully, with specific statements as to how the new findings should be applied clinically. Policy review UpToDate s policies and procedures are continuously reviewed in consultation with our Evidence- Based Medicine Advisory Group. Members of this group include Dr. Gordon Guyatt and Dr. Roman Jaeschke from McMaster University, Dr. Holger Schünemann from the Italian National Cancer Institute in Rome/McMaster University, and Dr. Yngve Falck-Ytter from Case Western Reserve University. 9.

10 Appendix 2: Hospital clinical knowledge systems and health outcomes study methods 26 Patient safety indicators The patient safety indicator (PSI) scores represent a set of measures on potential complications and adverse events on hospitalized patients. They were developed with a comprehensive literature synthesis, analysis of ICD-9 codes, and review by expert clinicians and implementation of risk adjustment and empirical studies. 48 The Agency for Healthcare Research and Quality (AHRQ) provides free, publicly available tools necessary to determine PSI scores and they are a widely used measure of patient safety. The AHRQ Patient Safety Indicators are used in at least 9 states for public reporting on hospital performance. The rationale is that hospitals that show good performance on these measures are likely to be providing good quality of care. There are 23 AHRQ PSIs that are relevant for provider-level (hospital-level) analysis. Seven, which are birth-related, are not relevant to a Medicare population and were thus excluded. Five require Cause of Injury or E-codes, which are not coded consistently across acute-care hospitals in the United States, and were therefore not included in the study. Complications Solucient has constructed a database containing normative, case-level data on a national level; it contains more than 21 million annual patient discharge levels. The case-level data include age, sex, race, payer, length of stay, clinical grouping (Diagnosis Related Groups or Refined Diagnosis Related Groups), comorbid conditions, and hospital identification information. The database permits analysis of hospital-level data for two general types of complications: 1) Conditions of concern in which there are outcomes that should not occur and thus represent substandard care (e.g. air embolism); 2) Expected complications (e.g. venous thrombosis) that (when occurring in higher than expected frequencies) may indicate opportunities for improving patient safety. Complication rates can be compared across hospitals after adjusting for differences in the severity of illness, geographic location, hospital size and teaching status, and community setting (urban versus rural). Thus, facilities are compared to other facilities with similar characteristics. The complications methodology has been extensively validated and used in many studies evaluating patient safety Mortality Risk-adjusted mortality is determined based upon normative comparisons using patient-level data to control for case mix and severity of illness. 52,53 Thus, patients are compared to other patients with similar characteristics and co-morbid conditions. The risk-adjusted mortality index has been extensively validated. 52,53 Length of stay Determination of average length of stay was based upon severity-adjusted diagnostic related groups (DRGs). 54,55 Severity-adjusted DRGs are intended to distinguish discharges that are clinically similar and require comparable resources (e.g. diagnostic, therapeutic, and nursing services). Severity of illness is based upon the patient s medical condition or evidence of physiologic decompensation. The DRG system is used to determine reimbursement by the Centers for Medicare and Medicaid Services and most private insurers. Eligibility Hospitals included in the Solucient database were required to have a Centers for Medicare and Medicaid Services (CMS) Medicare Provider Analysis and Review (MEDPAR) dataset for 2003 to 2004, CMS standard analytic files outpatient dataset for 2002 and 2003, and a CMS cost report for Hospitals were excluded from the Solucient database if a current Medicare cost report was unavailable, if they were specialty hospitals (e.g. children s, women s, psychiatric, substance abuse, rehabilitation, cardiac, orthopedic, long-term acute care), had fewer than 25 acute care beds, 500 total facility admissions, or 100 Medicare patient discharges in fiscal year 2004, or had Medicare average lengths of stay longer than 30 days. All UpToDate hospitals fulfilling these criteria were included in the analysis. Weighting strategy Hospitals that subscribed to UpToDate were compared to the Solucient database on bed size, teaching status, and geographic region. These are variables known to be associated with the primary outcome measures. Two weighting strategies were used to allow the model coefficients to be applicable to a national distribution of acute-care hospitals. Because all the outcome variables were based on dischargelevel observations, weighting for hospital volume was done to account for differences across hospital inpatient volume. 10.

11 Weighting by hospital class was done to derive model coefficients that would better reflect the national distribution of hospitals on bed size category, teaching status, and region. Thus, the analysis was performed without weighting, by weighting for discharge volume only, and by weighting for discharge volume and hospital class. Hospital class was categorized as small community (bed size 25-99), medium community (bed size ), or large community (bed size >250). Teaching hospitals were classified as standard teaching (bed size >250 and resident-to-bed ratio of >0.03 or total residency programs >3), or major teaching (bed size >400 and residents-to-beds ratio >0.25 and >10 residency programs, or resident-to-bed ratio >0.60 if fewer residency programs). Statistical analysis Hospital-specific performance on mortality, complications, and patient safety were represented by a z-score for each measure, while severity-adjusted length of stay (LOS) was based upon the Yale Refined Diagnosis-Related Group methodology. 28,56 Severity-adjusted LOS was represented in the models in units of days. Hospital-level, risk-adjusted mortality, complications, and PSI were based on two years of data (2003 and 2004) to increase the precision of hospital-level measures through larger sample sizes; LOS was based on 2004 data only. Data regarding the average hits per week were also matched to the Solucient database to determine whether associations with the primary outcomes correlated with the amount that UpToDate was used at each facility. Thus, UpToDate status was evaluated in two ways: 1) as a binary variable indicating whether a given hospital was using UpToDate and 2) as a continuous variable representing HPW. Hospitals that were not UpToDate users were assigned a zero HPW. The relationship between the primary outcome measures and HPW was presented graphically by converting changes in z-scores associated with UpToDate HPW to risk-adjusted rates. The PSI composite measure was created by calculating the average of the normalized z-scores for 11 individual AHRQ PSI indicators that were appropriate for a Medicare population aged 65 years and older, and did not depend on E-codes. As noted above, use of cause of injury codes, or E-codes, varies extensively from hospital to hospital, thus PSI measures that used E-codes were omitted. The measure-specific z-scores were calculated by subtracting the expected number from the observed number and dividing by the standard error. 57,58 The mortality, complications, and PSI z-scores, as well as the severity-adjusted LOS, were used as outcome variables in the analyses. Separate linear regression models were developed for each combination of outcome and weighting strategy. All models included adjustment variables described above to control for hospital bed size category, teaching status, and geographic region (i.e. Northeast, Midwest, South, and West). The primary predictor variables of interest were: 1) the UpToDate status of the hospital and, 2) in separate models, the average number of topic reviews per week (hits per week, or HPW). HPW refers to the average number of topic reviews that were viewed at each institution each week. All comparisons were between UpToDate hospitals versus all nonuptodate hospitals in the Solucient database. 11.

12 References 1. Alper BS, White DS, Ge B. Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care. Ann Fam Med 2005;3(6): Barrie AR, Ward AM. Questioning behaviour in general practice: a pragmatic study. BMJ 1997;315(7121): Bergus GR, Emerson M. Family medicine residents do not ask better-formulated clinical questions as they advance in their training. Fam Med 2005;37(7): Chambliss ML, Conley J. Answering clinical questions. J Fam Pract 1996;43(2): Cheng GY. A study of clinical questions posed by hospital clinicians. J Med Libr Assoc 2004;92(4): Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med 1985;103(4): Currie LM, Graham M, Allen M, Bakken S, Patel V, Cimino JJ. Clinical information needs in context: an observational study of clinicians while using a clinical information system. AMIA Annu Symp Proc 2003: D Alessandro DM, Kreiter CD, Peterson MW. An evaluation of information-seeking behaviors of general pediatricians. Pediatrics 2004; 113(1 Pt 1): Del Mar CB, Silagy CA, Glasziou PP, et al. Feasibility of an evidence-based literature search service for general practitioners. Med J Aust 2001;175(3): DeZee KJ, Durning S, Denton GD. Effect of electronic versus print format and different reading resources on knowledge acquisition in the third-year medicine clerkship. Teach Learn Med 2005;17(4): Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians clinical questions: obstacles and potential solutions. J Am Med Inform Assoc 2005;12(2): Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. BMJ 1999;319(7206): Ely JW, Osheroff JA, Ebell MH, et al. Obstacles to answering doctors questions about patient care with evidence: qualitative study. BMJ 2002;324(7339): Fozi K, Teng CL, Krishnan R, Shajahan Y. A study of clinical questions in primary care. Med J Malaysia 2000;55(4): Gorman P. Information needs in primary care: a survey of rural and nonrural primary care physicians. Medinfo 2001;10(Pt 1): Gorman P. Information needs in primary care: a survey of rural and nonrural primary care physicians. Medinfo 2001;10(Pt 1): Graber MA, Randles BD, Ely JW, Monnahan J. Answering clinical questions in the ED. Am J Emerg Med 2008;26(2): Green ML, Ciampi MA, Ellis PJ. Residents medical information needs in clinic: are they being met? Am J Med 2000;109(3): Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Acad Med 2005;80(2): Jerome RN, Giuse NB, Gish KW, Sathe NA, Dietrich MS. Information needs of clinical teams: analysis of questions received by the Clinical Informatics Consult Service. Bull Med Libr Assoc 2001;89(2): Kim GR, Bartlett EL, Jr., Lehmann HP. Information resource preferences by general pediatricians in office settings: a qualitative study. BMC Med Inform Decis Mak 2005;5: Koonce TY, Giuse NB, Todd P. Evidence-based databases versus primary medical literature: an in-house investigation on their optimal use. J Med Libr Assoc 2004;92(4): Leff B, Harper GM. The reading habits of medicine clerks at one medical school: frequency, usefulness, and difficulties. Acad Med 2006;81(5): Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med 2003;35(4): Swinglehurst DA, Pierce M, Fuller JC. A clinical informaticist to support primary care decision making. Qual Health Care 2001;10(4): Bonis PA, Rind DM, Pickens G, Foster D. Association of a Clinical Knowledge Support System with Improved Patient Safety, Reduced Complications and Shorter Length of Stay among Medicare Beneficiaries in Acute Care Hospitals in the United States. Int J Med Inform 2008:(in press). 27. Dickinson T, Riley J, Zabetakis PM. External validation of compliance to perfusion quality indicators. Perfusion 2004;19(5): Edwards N, Honemann D, Burley D, Navarro M. Refinement of the Medicare diagnosis-related groups to incorporate a measure of severity. Health Care Financ Rev 1994;16(2): Griffith JR, Alexander JA, Jelinek RC. Measuring comparative hospital performance. J Healthc Manag 2002;47(1): Griffith JR, Knutzen SR, Alexander JA. Structural versus outcomes measures in hospitals: a comparison of Joint Commission and Medicare outcomes scores in hospitals. Qual Manag Health Care 2002;10(2): Kirchheimer B. Hospitalists make a mark. Emerging specialty linked to shorter lengths of stay and improved mortality, but not to lower costs, Solucient finds. Mod Healthc 2006;Suppl:8-10, 2, 4 passim. 32. McDonagh KJ. Hospital governing boards: a study of their effectiveness in relation to organizational performance. J Healthc Manag 2006;51(6):377-89; discussion Nallamothu BK, Saint S, Ramsey SD, Hofer TP, Vijan S, Eagle KA. The role of hospital volume in coronary artery bypass grafting: is more always better? J Am Coll Cardiol 2001;38(7): Proenca EJ, Rosko MD, Dismuke CE. Service collaboration and hospital cost performance: direct and moderating effects. Med Care 2005;43(12): Safavi K. The measurement conundrum. J Healthc Manag 2006;51(5): Lucas BP, Evans AT, Reilly BM, et al. The impact of evidence on physicians inpatient treatment decisions. J Gen Intern Med 2004;19 (5 Pt 1): McDonald FS, Zeger SL, Kolars JC. Factors Associated with Medical Knowledge Acquisition During Internal Medicine Residency. J Gen Intern Med 2007;22(7): Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Quality & safety in health care 2003;12 Suppl 2:ii Wolters Kluwer Health ProVation Medical 800 Washington Avenue North Suite 400 Minneapolis, MN ProVationMedical.com

13 39. Thomas EJ, Studdert DM, Newhouse JP, et al. Costs of medical injuries in Utah and Colorado. Inquiry 1999;36(3): Krohn R. In search of the ROI from CPOE. J Healthc Inf Manag 2003;17(4): Menachemi N, Brooks RG. Reviewing the benefits and costs of electronic health records and associated patient safety technologies. Journal of medical systems 2006;30 (3): Adams M, Bates DH, Coffman GA, Everett W. Saving Lives, Saving Money: The imperative for computerized physician order entry in Massachusetts Hospitals. In: Massachusetts Technology Collaborative and the New England Healthcare Institute; Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4): Poon EG, Blumenthal D, Jaggi T, Honour MM, Bates DW, Kaushal R. Overcoming barriers to adopting and implementing computerized physician order entry systems in U.S. hospitals. Health affairs (Project Hope) 2004;23(4): McAlister FA, Straus SE, Guyatt GH, Haynes RB. Users guides to the medical literature: XX. Integrating research evidence with the care of the individual patient. Evidence- Based Medicine Working Group. JAMA 2000;283(21): Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328(7454): Guyatt G, Gutterman D, Baumann MH, et al. Grading strength of recommendations and quality of evidence in clinical guidelines: report from an american college of chest physicians task force. Chest 2006;129(1): AHRQ. Quality Indicators--guide to Patient Safety Indicators. version 21, Revision 3 ed: Agency for Healthcare Research and Quality; Iezzoni LI, Daley J, Heeren T, et al. Identifying complications of care using administrative data. Med Care 1994;32(7): Iezzoni LI, Davis RB, Palmer RH, et al. Does the Complications Screening Program flag cases with process of care problems? Using explicit criteria to judge processes. Int J Qual Health Care 1999;11(2): Weingart SN, Iezzoni LI, Davis RB, et al. Use of administrative data to find substandard care: validation of the complications screening program. Med Care 2000;38(8): DesHarnais S, McMahon LF, Jr., Wroblewski R. Measuring outcomes of hospital care using multiple risk-adjusted indexes. Health Serv Res 1991;26(4): DesHarnais SI, Chesney JD, Wroblewski RT, Fleming ST, McMahon LF, Jr. The Risk-Adjusted Mortality Index. A new measure of hospital performance. Med Care 1988;26(12): Fetter RB. Casemix classification systems. Aust Health Rev 1999;22(2):16-34; discussion Freeman JL, Fetter RB, Park H, et al. Diagnosisrelated group refinement with diagnosis- and procedure-specific comorbidities and complications. Med Care 1995;33(8): MacKenzie T. Patient classification systems: An evaluation of the state of the art Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD. Judging hospitals by severity-adjusted mortality rates: the influence of the severity-adjustment method. Am J Public Health 1996;86(10): Iezzoni LI, Shwartz M, Ash AS, Hughes JS, Daley J, Mackiernan YD. Using severity-adjusted stroke mortality rates to judge hospitals. Int J Qual Health Care 1995;7(2): Wolters Kluwer Health ProVation Medical 800 Washington Avenue North Suite 400 Minneapolis, MN ProVationMedical.com