CONSIDERABLE EFFORTS TO

Transcription

1 ORIGINAL ARTICLE Association Between Implementation of a Medical Team Training Program and Surgical Morbidity Yinong Young-Xu, ScD, MA, MS; Julia Neily, RN, MS, MPH; Peter D. Mills, PhD, MS; Brian T. Carney, MD; Priscilla West, MPH; David H. Berger, MD, MHCM; Lisa M. Mazzia, MD; Douglas E. Paull, MD; James P. Bagian, MD Objective: To determine whether there is an association between the Veterans Health Administration Medical Team Training (MTT) program and surgical morbidity. Design, Setting, and Participants: A retrospective health services study was conducted with a contemporaneous control group. Outcome data were obtained from the Veterans Health Administration Surgical Quality Improvement Program. The analysis included aggregated measures representing sampled procedures from 74 Veterans Health Administration facilities that provide care to veterans. Main Outcome Measures: The primary outcome measure was the rate of change in annual surgical morbidity rate 1 year after facilities enrolled in the MTT program as compared with 1 year before and compared with the non-mtt program sites. Results: Facilities in the MTT program (n=42) had a significant decrease of 17% in observed annual surgical morbidity rate (rate ratio, 0.83; 95% CI, ; P=.01). Facilities not trained (n=32) had an insignificant decrease of 6% in observed morbidity (rate ratio, 0.94; 95% CI, ; P=.11). After adjusting for surgical risk, we found a decrease of 15% in morbidity rate for facilities in the MTT program and a decrease of 10% for those not yet in the program. The risk-adjusted annual surgical morbidity rate declined in both groups, and the decline was 20% steeper in the MTT program group (P=.001) after propensity-score matching. The steeper decline in annual surgical morbidity rates was also observed in specific morbidity outcomes, such as surgical infection. Conclusion: The Veterans Health Administration MTT program is associated with decreased surgical morbidity. Arch Surg. 2011;146(12): Author Affiliations: National Center for Patient Safety, Department of Veterans Affairs (VA), White River Junction, Vermont (Drs Young-Xu, Mills, Carney, Berger, Mazzia, Paull, and Bagian, and Mss Neily and West); Department of Psychiatry, Dartmouth Medical School, Hanover, New Hampshire (Drs Young-Xu and Mills); Departments of Surgery, Baylor College of Medicine and Michael E. DeBakey VA Medical Center, Houston, Texas (Dr Berger); Departments of Biomedical Engineering, Medical School and College of Engineering, University of Michigan, and National Center for Patient Safety, Ann Arbor, Michigan (Dr Bagian). CONSIDERABLE EFFORTS TO reduce risk associated with operations have been promoted over the past decade. 1 Although surgical mortality dominates the headlines, it is, in fact, the more common nonfatal surgical complications that could provide the earliest signals of failures in the medical system. In a previous article, we 2 studied the effect that Medical Team Training (MTT) See Invited Critique at end of article had on surgical mortality rates. In the present study, we devoted our attention to surgical morbidity rates to contribute to the growing body of research regarding methods of reducing surgical errors. 3-8 In 2006, the Veterans Health Administration (VHA) implemented a nationwide MTT program to facilitate structured interactive communication in surgical care and improve surgical outcomes. The VHA MTT program requires preoperative briefings and postoperative debriefings guided by a checklist 9 and using cognitive aids. 10,11 The goal of this study was to compare the annual surgical morbidity rates for facilities that received the VHA MTT program with those that had not. CME available online at and questions on page 1345 METHODS INTERVENTION The MTT program 12,13 aims to improve the communications within the operating room (OR) and help the staff to work better as a team. It allows staff to identify potential trouble spots and any obstacles to a safe and successful operation through checklist-guided preoperative briefings and postoperative debriefings. The program began with 2 months of preparation and planning with an implementation team at the training VHA facility. This was followed by a 1-day learning session held on site. The facilities included were provided sample checklists and briefing/debriefing tools that facilities adapted for their needs. 9 The MTT program is based on crew resource management theory from aviation. 14 Everyone in the OR is trained to work as a team and challenge each other 1368

2 Table 1. Baseline Characteristics of 74 Facilities Characteristic MTT (n = 42) Non-MTT (n = 32) P Value Observed annual surgical morbidity rate per 1000 procedures per year a Mean (SD) 90 (45) 81 (41) Median (range) 97 (5-241) 89 (3-160).49 Risk-adjusted annual surgical morbidity rate per 1000 procedures per year a Mean (SD) 97 (23) 88 (21) Median (range) 96 (41-169) 93 (25-128).14 Observed mortality rate per 1000 procedures per year a Mean (SD) 16 (8) 13 (8) Median (range) 16 (0-34) 15 (0-27).15 Risk-adjusted annual surgical mortality rate per 1000 procedures per year a Mean (SD) 16 (7) 14 (5) Median (range) 17 (3-35) 13 (2-24).10 VASQIP surgical volume, cases per facility baseline year a Mean (SD) 754 (469) 792 (401) Median (range) 616 ( ) 845 ( ]).55 No. (%) High-complexity facility b 52 (22) 63 (20).41 Urban facility b 81 (34) 84 (27).70 Abbreviations: MTT, Medical Team Training; VASQIP, Veterans Health Administration Surgical Quality Improvement Program. a For variables that are not normally distributed, we used a nonparametric method (Mann-Whitney) for the testing equality between trained and nontrained groups. b For comparison of proportions (high-complexity facility and urban facility), we used the 2 test. when they identify safety risks. After the learning session, 4 quarterly follow-up telephone interviews are conducted with the team to provide support, coaching, and assessment of implementation of the MTT program. Details of the MTT program have been published. 2 STUDY DESIGN This was a retrospective health services cohort study using a contemporaneous control group. The VHA Surgical Quality Improvement Program (VASQIP) annual surgical morbidity rates from 3 fiscal years 2006, 2007, and 2008 were used. EXCLUSION CRITERIA The training list contained 130 facilities. Facilities on the training list but without all 3 years of VASQIP data were excluded from the analysis (n=14). Facilities trained during the pilot phase of the study were also excluded (n=10). Facilities trained in 2008 (n=32) were excluded because data for the 2009 fiscal year were unavailable. In total, 74 facilities were analyzed. OUTCOME MEASURES Considered as the criterion standard for measuring surgical quality and adopted by the American College of Surgeons, 15 VASQIP provides observed and risk-adjusted 30-day morbidity rates for major noncardiac operations Surgical specialties included general, orthopedic, urology, vascular, neurosurgery, otolaryngology, noncardiac thoracic, and plastic. Current Procedural Terminology 18(p495) codes of procedures with known low morbidity and mortality rates or transurethral resections of the prostate, transurethral resections of the bladder tumor, and herniorrhaphies exceeding the limit of 5 per week were excluded. Using VASQIP data, we included annual surgical morbidity rates as our primary end points for each of the 74 facilities for fiscal years from 2006 to Because MTT is a facility-level intervention, aggregated morbidity rates at the facility level were outcome measures. Although not studied as an outcome here because it was the focus of a previous study, 2 mortality data were used as a covariate in statistical modeling. Unless otherwise stated, all morbidity and mortality rates were annual, and both observed and risk-adjusted rates were studied. All years were meant to be VHA fiscal years. For example, year 2008 indicates fiscal year 2008 (from October 1, 2007, to September 30, 2008). The VASQIP methods have been described in detail in other studies Detailed definitions and descriptions of outcomes can be found in the eappendix ( To examine baseline characteristics of the MTT and non- MTT facilities, we compared the following: rural or urban status, complexity, VASQIP surgical volume, and baseline annual surgical morbidity as well as mortality rate (Table 1). The VHA 2005 Complexity Model 20 categorizes VHA facilities into 3 levels: level 1 is high complexity; level 2, medium complexity; and level 3, low complexity. These designations are based on a composite involving the number of patients evaluated, patient risk, number of physician specialists, teaching status, research funding, and intensive care unit capability. The VHA urban/rural/highly rural classification is based partly on census tract and partly on population density. Facilities located in US census tracts designated as urban are considered urban. All others are considered rural, except for facilities located in a county with a population density of less than 7 people per square mile, which are considered highly rural. 21 STATISTICAL ANALYSIS Annual surgical morbidity rate was defined as the number of VASQIP surgical morbidities divided by the number of procedures during the fiscal year. The primary outcome measure was change in morbidity rate from 2006 (1 year before facilities enrolled in the MTT program) to 2008 (1 year after facilities enrolled in the MTT program). Continuous variables not distributed normally were compared using the Mann-Whitney test; their medians and ranges, as well as means are presented. Pearson 2 tests were used to compare proportions (Table 1 and Table 2). Comparisons of morbidity rates be- 1369

3 Table 2. Characteristics at Baseline by Propensity Score Tertiles According to Medical Team Training Program Selection Tertiles of Propensity Score, Mean (Range) a Characteristic 0.49 ( ) 0.67 ( ) 0.87 ( ) Observed annual surgical morbidity rate per 1000 procedures per year Nontrained (No.) 75 (15) 80 (9) 114 (8) Trained (No.) 72 (9) 80 (19) 115 (14) P value Risk-adjusted annual surgical morbidity rate per 1000 procedures per year Nontrained Trained P value Observed mortality rate per 1000 procedures per year Nontrained Trained P value Risk-adjusted mortality rate per 1000 procedures per year Nontrained Trained P value Mean No. of procedures sampled per facility Nontrained Trained P value Urban facilities, % Nontrained Trained P value High-complexity facilities, % Nontrained Trained P value a Propensity analysis was based on the observed and risk-adjusted surgical mortality rates, the observed and risk-adjusted surgical morbidity rates, the average number of sampled procedures per facility, hospital complexity, and urbanicity. Table 3. Event Rates Before and After, According to MTT Rate per 1000 Operations MTT (n = 42) Non-MTT (n = 32) Rate Ratios (MTT vs Non-MTT) Before After P Value a Before After P Value a Rate Ratio b (95% CI) Morbidity c ( ).001 d Risk-adjusted morbidity d e 1.20 ( ).001 d Specific complications DVT e ( ).04 e PE ( ).12 DVT PE e ( ).03 e Superficial surgical infection c ( ).03 e Deep wound infection e ( ).05 e All infections c ( ).005 c Abbreviations: DVT, deep venous thrombosis; MTT, Medical Team Training; PE, pulmonary embolism. a Before-and-after comparisons were conducted using the paired Wilcoxon signed rank test. P values have been rounded. b Rate ratios were estimated using propensity score matched multivariate generalized estimating equation analyses. c Significant at the.01 level. d Significant at the.001 level. e Significant at the.05 level. P Value a fore (2006) and after (2008) were made separately for MTT and non-mtt groups, using paired Wilcoxon signed rank test (Table 3). Multivariable Poisson generalized estimating equations 22,23 (GEEs) were used to assess MTT associations with outcome while adjusting for secular trends as well as propensity scores (Table 3). Rate ratios (RRs) and accompanying 95% CIs were calculated to represent the strength of association between MTT intervention and morbidity rates. The GEE method was used to account for the longitudinal nature of the yearly data. We controlled for baseline characteristics that included complexity, surgical volume, urbanicity, and baseline morbidity and mortality rates. Along 1370

4 with the main effect of MTT and time (unit in years), an interaction term, MTT time, was entered into the model to study whether the trajectories of the 2 groups differed. We compared not only MTT program facilities with those not yet trained but also morbidity rates in the same facility after receiving vs before receiving MTT. As a result, some facilities served as their own controls as well as controls for others. PROPENSITY SCORE Because MTT was not randomly assigned in this study, potential confounding and selection biases were accounted for by developing a propensity score for MTT. 24,25 The propensity score was based on the following variables measured at baseline: morbidity and mortality rates, the average number of sampled procedures per facility, hospital complexity, and urbanicity. These were summarized as 1 propensity score through a full nonparsimonious logistic model. For each facility, a propensity score was calculated and facilities then were stratified by their propensity scores into 3 strata. Within these strata, covariates in MTT and non- MTT groups were similarly distributed, and stratifying by propensity score could remove more than 90% of the overt bias due to the covariates used to estimate the score. 26 Once groups were stratified by propensity score, they were again separated into MTT and non-mtt groups to determine whether there were differences in baseline variables, such as morbidity rates, to suggest an imbalance. Propensity scores could not remove hidden biases except to the extent that unmeasured variables were correlated with the measured covariates used to compute the score Propensity score matching was used to select non-mtt program facilities that were similar to MTT program facilities with respect to propensity score and other covariates, thereby matching on many confounders simultaneously. 27 As a result, in each propensity score stratum defined by this procedure, the covariates were balanced and the assignment to MTT could be considered random. At that time, within each stratum, a full Poisson GEE analysis was performed to compute the effect of the MTT program. The total effect of the MTT program was finally obtained as a summary of the effect estimate of each stratum. This provided a valid estimate of the MTT program effect because the GEE analyses compared facilities with similar baseline characteristics. All analyses were performed using commercial software (STATA, version 10.0; StataCorp, College Station, Texas). All P values were 2-sided at a significance level of.05. The study was approved by the Research and Development Committee at the VA Medical Center in White River Junction, Vermont, and considered exempt by the Dartmouth College institutional review board. RESULTS A total of sampled procedures from 74 facilities were analyzed. The MTT program was implemented in 42 facilities (57%), and 32 facilities not-yet trained (43%) served as the contemporaneous control group. Before-and-after analyses were performed for the MTT group and the non- MTT group, and the changes in the 2 groups were compared. The MTT program was implemented at 42 facilities in Thirty-two facilities were not trained in the MTT program during The baseline (before) for all 74 facilities was Because 2008 (after) was the year following the 2007 implementation year for the MTT program facilities, we chose 2008 as the follow-up measure for all 74 facilities. The 2 groups did not differ significantly (Table 1) in morbidity or mortality rates, surgical volume, surgical complexity, 24 or urbanicity. 25 Because facilities that underwent MTT were not randomly selected, we tried to correct potential imbalances between the trained and the nontrained groups through propensity score matching. Table 2 displays the 3 groups of facilities stratified by propensity scores for each baseline variable. Mean MTT selection propensity scores ranged from 0.54 to 0.78 across propensity tertiles, with excellent discrimination between MTT and non-mtt groups (C statistic=0.79). The distribution of key potential confounders morbidity rates at baseline was similar within propensity tertiles for facilities with and without the MTT program. This indicates that none of the 3 propensity score strata showed significant differences in morbidity rates between the groups at baseline. In short, we were not able to discern any overt selection bias in our model after matching on propensity score. The change in morbidity rate from baseline was examined: facilities in the MTT program (n=42) had a statistically significant decrease of 17% in observed morbidity rate (RR, 0.83; 95% CI, ; P=.01). Facilities not in the program (n=32) had a decrease of 6% in observed morbidity; however, this was not statistically significant (RR, 0.94; 95% CI, ; P=.11). After risk adjustment, both groups showed statistically significant improvement a decrease of 15% for those in the MTT program and a decrease of 10% for those not yet in the MTT program (Table 3) unlike the findings without risk adjustment. The magnitude of the difference between trained and untrained groups was also smaller after risk adjustment (a 5 percentage point difference vs 11 percentage point difference). To put this in another way, 88% of the MTT program facilities (37 of 42) improved their risk-adjusted morbidity rates vs 69% of the non-mtt group (22 of 32; P=.04). Although risk-adjusted morbidity rates declined in both groups, the decline was 20% steeper in the MTT group (RR, 1.20; 95% CI, ; P=.001) after further adjustment for morbidity and mortality rates, procedure volumes, and propensity scores (Table 3). Wilcoxon matched-pairs signed rank tests were performed for other outcome measures, although only total morbidity was risk adjusted by VASQIP. Nevertheless, significant drops in event rates were observed in all but one outcome (pulmonary embolism) for the MTT group. Decreases were also observed in the non-mtt group, albeit smaller, and none was statistically significant. Because of the low event rates for deep venous thrombosis and pulmonary embolism, and because they were clinically linked, additional analysis was performed in which these morbidities were combined as a single outcome, and combined infection end points were also analyzed (Table 3). Propensity score matched, surgical risk adjusted GEE analyses were similarly applied to specific nonfatal complications. Because we observed a decline trend in both the MTT and non-mtt groups, we also found that the decline in the MTT group was steeper than that in the non-mtt group by 15% (RR,1.15) to 28% (RR, 1.28; Table 3). COMMENT The VHA MTT program was associated with a statistically significant reduction in the annual surgical mor- 1371

5 bidity rate. Meanwhile, the ongoing VASQIP program uses a robust system of data feedback, support, and follow-up to address processes of care in each facility. Because of this, the results described in this study occurred within the larger context of the VASQIP s active quality improvement program. Thorough preparation before an operation and effective communication during the operation are promising methods of reducing surgical errors. 3-8 However, studies 3-8 in this area have been lacking in measurable medical outcomes (eg, length of survival, as well as rates of complications and recurrences), and many studies lack a control group or baseline performance data. Although checklists have been shown 7,8 to significantly reduce surgical complications, the VHA MTT program focuses more broadly on teamwork and on the use of briefings as the main tool to enhance proactive communication among the clinical staff. A checklist, although important, is only one component of the overall strategy to improve communication, as a static checklist cannot anticipate and explicitly include every possible factor. Briefings aided by a checklist, however, are able to facilitate and improve communication throughout an operation, which is a dynamic process that invariably has unforeseen complications. Another difference between the VA MTT program and other crew resource management based programs 14 is the 2-month preparation and planning phase in which MTT faculty work with an implementation team consisting of surgical staff at the facility to assess problem areas in the OR. During this time, the OR staff gains a deeper understanding of local communication problems at their facility that could be addressed by MTT. In addition, the 12-month follow-up helps the OR staff to implement the communication tools learned in the training, solve unanticipated problems, and provide data to the MTT faculty regardingtheirprogress. Finally, quarterlytelephonecallskeep pressure on the OR teams to continue to improve and disseminate the program throughout their services. Further research is needed to determine which aspects of the VA MTT program account for the change in surgical morbidity. Details regarding degrees of briefing and debriefing were collected during the 4 follow-up telephone interviews conducted during the year after the MTT program was instituted. 2 Based on these interviews, we hypothesize that preoperative checklist driven briefings provide an opportunity to correct problems before they surface. Surgical teams have shared stories, such as discovering that a patient required cardiac clearance, and others identified during the briefing the need for additional equipment or implants. 2 Teams also shared the value of voicing problems in the debriefing. Examples included fixing broken equipment or instruments, ordering extra or back-up sets of instruments to prevent intraoperative delays, and improving collaboration with the radiology service for quicker response times. The resolution of such issues likely also prevented adverse events. This study has several limitations. First, it was not randomized; consequently, unmeasured potential confounders are likely to exist. One concern is the imbalance between the intervention (MTT) and control (non-mtt) groups at baseline. Another is potential bias in the formation of study groups. For example, the first facilities to complete the MTT program may also be those with the greatest likelihood to improve. We addressed these concerns because propensity score matching approximates an unbiased design. We grouped the facilities according to their propensity scores based on their baseline characteristics. As a result, some groups had limited sample sizes. We then conducted analyses within these groups to control for potential confounders. Before propensity score matching, the difference between the MTT and non-mtt groups in reduction of the risk-adjusted morbidity rate was 50% (15% in the MTT group vs 10% in the non-mtt group). After using propensity score matching, the estimated difference was 20% (RR, 1.20). Our limited sample size and observational study design might partially or fully account for the observed association; thus, we caution against a simplistic causal interpretation of our model. However, studies with different designs, populations, and research groups appear to have arrived at a similar estimate of the impact of surgical checklists on morbidity. 7,8 We adopted the more flexible and versatile longitudinal GEE model to analyze the data because it could ameliorate some of the shortfalls in study design. For example, each facility served as its own control, thus enabling us to remove some extraneous, but unavoidable, sources of variability among the facilities, such as location, size, and structure. Although the VHA introduced the MTT program because facilities needed to improve communication, it is nevertheless possible that facilities could have started to implement some aspects of the MTT program before their enrollment. The best way that we could address this limitation was to use facilities as their own controls in the analysis and thus remove the effect of heterogeneity from this source. Because many factors can reduce surgical morbidity, the inclusion in our design of a contemporaneous control group from the study period that was similar to the groups at the MTT facilities after being matched on propensity scores should have lessened the potential for confounding due to existing secular trends. We observed a general secular trend of decreasing surgical complications in the VHA during our study period (2006 vs 2008) for non-mtt facilities. Many contributing factors are likely: the active improvement program of VASQIP, the cost of complications in an era of health care cost containment, continuous efforts to improve the quality and safety of care for veterans, modified indications for operations, the advancement of medical science and medical training, and many other efforts to reduce medical errors (eg, better system design, equipment design, and medication design [packaging, labeling, administering, and tracking]). We found that the MTT program, beyond these general efforts and secular trend, resulted in significant additional reduction in surgical morbidity. Finally, the applicability of this study to the general population may be limited because VHA patients have been found 16 to differ from patients in the private sector. In conclusion, participation in the VHA MTT program, which emphasizes communication and teamwork during an operation through checklist-driven brief- 1372

6 ings and debriefings, is associated with lower surgical morbidity. Accepted for Publication: May 26, Correspondence: Yinong Young-Xu, ScD, MA, MS, National Center for Patient Safety, 215 N Main St, White River Junction, VT ([email protected]). Author Contributions: Dr Young-Xu and Ms Neily had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Neily, Mills, Berger, Paull, and Bagian. Acquisition of data: Carney, West, Mazzia, and Paull. Analysis and interpretation of data: Young-Xu and Berger. Drafting of the manuscript: Young- Xu. Critical revision of the manuscript for important intellectual content: Neily, Mills, Carney, West, Berger, Mazzia, Paull, and Bagian. Statistical analysis: Young-Xu. Administrative, technical, and material support: Mills, West, Mazzia, Paull, and Bagian. Study supervision: Neily, Berger, Mazzia, Paull, and Bagian. Financial Disclosure: None reported. Funding/Support: This material is the result of work supported with resources and the use of facilities at the VHA National Center for Patient Safety, Ann Arbor, Michigan, and Field Office in White River Junction, Vermont, and the Michael E. DeBakey VA Medical Center, Houston, Texas. Role of the Sponsors: The design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript was done as part of the work of VHA employees and there was no other sponsor or funding agency. Disclaimer: The opinions expressed are those of the authors and not necessarily those of the Department of Veterans Affairs or the United States government. Online-Only Material: The eappendix is available at http: // Additional Contributions: Members of the VA Surgical Quality Data Use Group served as scientific advisors and performed critical review of data use and analysis presented in this article. Shoshana Boar, MS, supported the VHA Medical Team Training program by coordinating all the training programs, scheduling, and taking notes during all follow-up interviews, and Lori Robinson, RN, MS, served as one of the instructors for the VHA Medical Team Training program. Mss Boar and Robinson work for the Department of Veterans Affairs and received no other compensation related to this study. REFERENCES 1. Institute of Medicine. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; Neily J, Mills PD, Young-Xu Y, et al. Association between implementation of a medical team training program and surgical mortality. JAMA. 2010;304(15): Mazzocco K, Petitti DB, Fong KT, et al. Surgical team behaviors and patient outcomes. Am J Surg. 2009;197(5): Davenport DL, Henderson WG, Mosca CL, Khuri SF, Mentzer RM Jr. Riskadjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. J Am Coll Surg. 2007; 205(6): Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004;13(suppl 1):i85-i Awad SS, Fagan SP, Bellows C, et al. Bridging the communication gap in the operating room with medical team training. Am J Surg. 2005;190(5): Haynes AB, Weiser TG, Berry WR, et al; Safe Surgery Saves Lives Study Group. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360(5): de Vries EN, Prins HA, Crolla RM, et al; SURPASS Collaborative Group. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010; 363(20): Paull DE, Mazzia LM, Izu BS, Neily J, Mills PD, Bagian JP. Predictors of successful implementation of preoperative briefings and postoperative debriefings after medical team training. Am J Surg. 2009;198(5): Mills PD, DeRosier JM, Neily J, McKnight SD, Weeks WB, Bagian JP. A cognitive aid for cardiac arrest: you can t use it if you don t know about it. Jt Comm J Qual Saf. 2004;30(9): Neily J, DeRosier JM, Mills PD, Bishop MJ, Weeks WB, Bagian JP. Awareness and use of a cognitive aid for anesthesiology. Jt Comm J Qual Patient Saf. 2007; 33(8): Dunn EJ, Mills PD, Neily J, Crittenden MD, Carmack AL, Bagian JP. Medical team training: applying crew resource management in the Veterans Health Administration. Jt Comm J Qual Patient Saf. 2007;33(6): Neily J, Mills PD, Lee P, et al. Medical team training and coaching in the veterans health administration; assessment and impact on the first 32 facilities in the programme. Qual Saf Health Care. 2010;19(4): Musson D, Helmreich RL. Team training and resource management in health care: current issues and future directions. Harvard Health Policy Review. 2004;5(1): American College of Surgeons, National Surgical Quality Improvement Program. About ACS NSQIP, history of the ACS NSQIP. Accessed September 20, Khuri SF, Henderson WG, Daley J, et al; Principal Site Investigators of the Patient Safety in Surgery Study. The patient safety in surgery study: background, study design, and patient populations. J Am Coll Surg. 2007;204(6): Khuri SF, Daley J, Henderson W, et al. The National Veterans Administration Surgical Risk Study: risk adjustment for the comparative assessment of the quality of surgical care. J Am Coll Surg. 1995;180(5): Khuri SF, Daley J, Henderson W, et al; National VA Surgical Quality Improvement Program. The Department of Veterans Affairs NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. Ann Surg. 1998; 228(4): Khuri SF. The NSQIP: a new frontier in surgery. Surgery. 2005;138(5): Veterans Healthcare Administration National Leadership Board Human Resources Committee Facility complexity model /vhapublications. Accessed December 30, West AN, Lee RE, Shambaugh-Miller MD, et al. Defining rural for veterans health care planning. J Rural Health. 2010;26(4): Diggle PJ, Liang KY, Zeger SL. Analysis of Longitudinal Data. Oxford, England: Clarendon Press; McCullagh P, Nelder JA. Generalized Linear Models. 2nd ed. London, England: Chapman & Hall; Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1): doi: /biomet/ Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med. 1997;127(8, pt 2): Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc. 1984;79: Rosenbaum PR, Rubin DB. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am Stat. 1985;39: