Heart Surgery Ratings Background and Methodology August 2011

Heart Surgery Ratings Background and Methodology August 2011 Background The Society of Thoracic Surgeons (STS) is a not-for-profit organization representing more than 6,000 surgeons, researchers, and allied health professionals worldwide involved in heart, lung, esophageal and other thoracic surgical procedures. Founded in 1964, the mission of STS is to enhance the ability of cardiothoracic surgeons to provide the highest quality patient care through education, research and advocacy. The Society continues its commitment to surgical quality through collaborations with government and health care quality organizations, including the National Quality Forum, the Agency for Healthcare Research and Quality, and the Surgical Quality Alliance. The STS National Database is a clinical data registry for cardiothoracic surgery, including three components: the Adult Cardiac Surgery Database, the General Thoracic Surgery Database, and the Congenital Heart Surgery Database. The STS National Database includes more than 1,300 participating sites (hospitals or surgical groups) and more than 3,500 cardiothoracic surgeons. The component databases have participants from 49 states and Washington, DC, along with one Canadian program. Founded in 1989, the Adult Cardiac Surgery Database is the largest cardiothoracic surgery outcomes and quality improvement program in the world. This database component contains more than 4.5 million surgical records, representing approximately 94 percent of all adult cardiac surgery centers throughout the U.S. More than 1,000 surgical groups, representing almost 3,000 surgeons, add new patient data four times each year. The Adult Cardiac Surgery Database includes data from a number of cardiothoracic surgical procedures, including coronary artery bypass grafting (CABG), valve repairs or replacements, and CABG in combination with valve repair or replacement. The Heart Surgery Ratings reflect only isolated CABG operations CABG performed in isolation, excluding those in which CABG was done in combination with other procedures. This is done in order not to mix apples and oranges, as combined procedures often carry a substantially higher inherent risk. The balance of this document refers only to these isolated CABG operations which constitute relatively homogeneous procedures that have typically been used to assess cardiac surgery provider performance. Consumers Union of U.S., Inc., 2011 1

Duke Clinical Research Institute (DCRI) is the data warehouse and analysis center for the STS Adult Cardiac Surgery Database. On behalf of STS, DCRI develops group-specific reports that provide analysis of groups adult cardiac surgery outcomes. These reports benchmark each group s data against regional and national outcomes, and allow surgeons to evaluate practice patterns and target specific areas for clinical practice improvement. These Heart Surgery Ratings are made available to consumers by special agreement between The Society of Thoracic Surgeons and Consumers Union. Each surgical group that submits data to STS was asked for permission to share their CABG results with Consumers Union for public reporting to consumers. Approximately one-third of the surgical groups with complete data submitted to the STS Adult Cardiac Surgery Database agreed to participate in this initiative and have their results included here. In the future, we will update these ratings to include additional groups that consent, as well as to provide the most up-to-date data available. Some surgical groups provide data on CABG surgical quality to other organizations in their state or region, and several states have publicly-available reports. However, none of these reports is as comprehensive as the STS data. Measuring Surgical Quality In 2004, using materials from the STS National Database, the National Quality Forum (NQF) published 21 performance measures for CABG surgery, the first set of national standardized performance measures to assess the performance and outcomes of adult cardiac surgery. STS selected measures from this list to represent both surgical processes (how well surgeons adhere to established best practices) and outcomes (how patients fare), and to represent quality of care before, during, and after surgery. The STS ratings assess quality as a composite of 11 measures, grouped into four quality domains: patient survival, absence of complications, optimal surgical technique, and recommended medications, shown in the table below. 1 Measures of CABG surgical quality, in four quality domains Outcome measures Process measures 1. Patient survival: Risk-adjusted 30- day operative mortality. 2. Absence of complications: Riskadjusted morbidity. Patients must avoid all 5: a. Kidney failure b. Deep sternal wound infection c. Re-operation for any cause d. Stroke e. Prolonged breathing support 3. Optimal surgical technique: Use of at least one internal mammary artery graft. 4. Recommended medications: Patient must receive all 4: a. Preoperative beta-blocker b. Discharge aspirin c. Discharge beta-blocker d. Discharge antilipid therapy 1 For more detailed discussion of the development of this measure set, see Shahian DM, Edwards FH, Ferraris VA et al. Quality measurement in adult cardiac surgery: Part 1 Conceptual framework and measure selection. Annals of Thoracic Surgery, 2007, 83: pp. 3-12. Consumers Union of U.S., Inc., 2011 2

Ratings compare each group s performance to the average of all reporting surgical groups. Because survival (mortality) and complications (morbidity) are outcomes that depend on how sick patients are, these rates are statistically adjusted for the average risk of each group s patients. Recommended medications and use of the optimal surgical technique should be made available to all eligible patients, and are not risk-adjusted. Note that surgical volume (the number of operations performed by a group) is not included as a quality measure. While volume is sometimes considered a measure of surgical quality in other contexts, research demonstrates only a weak, if any, relationship between volume and outcomes for CABG. Further, volume is most useful as a quality indicator only as a surrogate for outcomes when true outcome data are unavailable. As the outcome measures of patient survival and absence of complications are included in the STS quality measurement approach, surgical volume is not included. Despite this, the number of isolated CABG operations reported by each group plays an important role in the statistical analysis, as described later. 1. Patient survival This Rating is an estimate of the probability that a patient will be discharged from the hospital and still be alive 30 days after surgery, after adjusting for the risk profile of the group s patients. This is the inverse of the more commonly-reported operative mortality rate, and is used to make the direction similar among the four measures in the composite (higher numbers are better). The average risk-adjusted CABG survival rate for all groups reporting to STS was 98 percent and the survival rates for most groups are tightly clustered around this value, reflecting the very good survival record for CABG surgery overall. Even the best overall surgical groups have average mortality performance, because overall average survival is so high. There are two relevant measures of mortality used in quality measurement. The one currently used by STS, operative mortality, is based on the number of patients who die at any time within the hospital stay, or are discharged and die within 30 days post-surgery, regardless of where they are. An alternative measure is in-hospital mortality, based on the proportion of patients who die at any time during the hospitalization for their surgery. There are strengths and weaknesses with the use of either measure of mortality. For example, it is more difficult to accurately monitor the survival of patients after discharge, leading to increased data validity concerns in measuring operative mortality. On the other hand, in-hospital mortality fails to capture deaths among patients who die of causes related to their surgery, but after they are discharged from the hospital. 2. Absence of complications There are five major potential complications that are of the greatest concern following CABG surgery: kidney failure, deep sternal wound infection, re-operation for any cause, stroke, or required prolonged breathing support. Because absence of all these complications is the desired outcome, they are considered as a bundle. This Rating is an estimate of the probability that an individual patient will experience none of these five complications during their hospitalization, after adjusting for the risk profile of the group s patients. Consumers Union of U.S., Inc., 2011 3

In all isolated CABG surgeries reported to STS, a risk-adjusted average of 85 percent of patients avoided all five major complications. 3. Optimal Surgical Technique There are many accepted variations of CABG surgical procedures. However, use of at least one internal mammary artery (IMA) graft during coronary bypass surgery has been universally recognized to impact long-term survival. This Rating is an estimate of the probability that a patient will have receive at least one IMA graft. On average, IMA grafts were used in 95 percent of surgeries reported to STS nationwide. 4. Recommended Medications Accepted best practice for cardiac bypass patients includes the administration of 4 types of medications: beta-blockers before surgery, and aspirin, beta-blockers, and lipid-lowering therapy after surgery. Use of each of these medications is supported by extensive research either specifically related to CABG surgery or to the secondary prevention of coronary disease. This Rating is an estimate of the probability that a patient will receive all four of these medications before and after surgery. Of all surgical groups reporting to STS, an average of 77 percent of patients received all four medications. Data Collection and Validation Surgical groups that choose to participate in the STS Adult Cardiac Surgery Database agree to update their data four times per year (the four annual data harvests ) and to participate in data validation efforts. Some 75 data elements are collected for each eligible surgical patient. As the data are subjected to rigorous validity checks, surgical groups correct any data discrepancies and complete missing data, and ultimately sign-off on the accuracy and completeness of their data. In addition, in 2006 STS established an independent, on-site audit program, conducted by the Iowa Foundation for Medical Care, Iowa and Illinois Medicare Quality Improvement Organization. Each year, a randomly selected sample of surgical groups is identified to participate in an on-site audit. Data submitted by each selected surgical group are compared to medical charts, electronic health records, and surgical logs, and the group is provided with detailed feedback about any discrepancies or other concerns that are identified. If data are missing on the delivery of a medication, the medication is assumed not to have been administered. Similarly, if there is missing data for mortality or any of the morbidities (post-surgical complications), the data are assumed to be missing because they did not occur. In each of these cases, missing data are imputed to be no. The analysis leading to the Heart Surgery Ratings is based only on adult patients (at least 18 years of age) who received an isolated coronary artery bypass graft (CABG) operation during 2009. Patients who received other types of cardiothoracic procedures, or CABG in combination with other procedures, are excluded. Consumers Union of U.S., Inc., 2011 4

Statistical Analysis of Outcomes Data and measures. For each surgical group, the percentage success in each of the four quality domains (survival, absence of complications, surgical technique, and medications) is calculated: Patient survival is the percentage of eligible patients who were discharged from the hospital alive and who survived at least 30 days post-surgery. Absence of complications is the percentage of patients who experienced none of the five major complications during their hospitalization. Optimal surgical technique is the percentage of patients who received at least one IMA graft. Recommended medications is the percentage of patients who received all four medications. Patients who do not survive until hospital discharge are not included in calculations for the three discharge medications. The surgical technique (IMA) measure excludes patients with a previous CABG surgery. Patients with a previous cerebrovascular accident are excluded from the stroke measure. Otherwise, patients who are not eligible for particular processes are assumed to be proportionally distributed among surgical groups, and no further adjustments are made. A risk score is calculated for each patient, using the existing STS CABG mortality and morbidity models 2, and an average risk is calculated for all the patients within a surgical group. Statistical method. Using the four percentages and average risk described above, estimates of the true probability of success in each domain are calculated through a multivariate (all measures estimated together in one model) hierarchical (patients clustered within surgical groups) logistic Bayesian regression model. 3 The risk adjustment for survival and complications leads to an estimate of the probability of success for the surgical group if the group treated patients having a risk profile similar to the national average risk for all STS groups. The hierarchical model incorporates information from all groups in the estimation of each group s results. Group-specific probabilities are estimated as a combination of the group s success rates and the mean rate. For groups with smaller numbers of CABG cases in the analysis, the estimation procedure borrows strength from the national mean and the 2 Shahian DM, O'Brien SM, Filardo G, Ferraris VA, Haan CK, Rich JB, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1 Coronary artery bypass grafting surgery. Annals of Thoracic Surgery 2009 Jul; 88(1 Suppl): pp. S2-22. 3 A detailed discussion of the statistical models and methods can be found in O Brien SM, Shahian DM, Delong ER, et al. Quality measurement in adult cardiac surgery: Part 2 Statistical Considerations in composite measure scoring and provider ratings. Annals of Thoracic Surgery 2007; 83: pp.13-26. Consumers Union of U.S., Inc., 2011 5

mean comprises a larger part of the group s estimated probability. For groups with a larger number of CABG cases, the mean plays a smaller role. Groups with a smaller number of cases are more difficult to differentiate from the mean and are more likely to receive average scores. Composite score. Following the estimation of success probabilities (risk-adjusted for patient survival and absence of complications) for the four domains, the composite score is calculated as the weighted sum of these rates. The weight for a domain is approximately equal to the reciprocal of the between-groups standard deviation for that domain. Consequently, a one percentage point difference in survival has the same impact on the composite score as does an 8 percentage point difference in complications, an 11 percentage point difference in surgical technique, or a 28 percentage point difference in recommended medications. This weighting scheme avoids the need to make subjective assessments of the relative importance of various measures. (The same is true for the all-or-none scoring used for medications and avoidance of complications.) Star Ratings Scores for the composite and the four domains are presented in 3 tiers: better than average, average, and worse than average, represented by 3, 2, and 1 stars respectively, and are based on Bayesian tests at the 98 percent probability level. That is, there is a 98 percent probability that a 3-star provider is better than the average of all providers, and a 98 percent probability that a 1-star provider is worse than average. Note that this Bayesian data analysis method lends itself to this direct probability interpretation of results, while more traditional methods rely on statistical significance, which has a less direct probabilistic interpretation. As a result of the small number of CABG cases for some providers and the demanding standard for these statistical tests, most surgical groups fall in the two-star range. These groups performance was either very close to the STS national mean, or else the number of patients was too small to reliably differentiate these groups from the mean. Numerous statistical models and criteria were investigated, and the methods chosen appeared to provide the best balance between sensitivity and specificity. Since most surgical groups have very high survival rates, other CABG quality reporting programs based solely on survival (or mortality) have difficulty discriminating among groups. By incorporating additional measures in the composite, the STS approach provides better discrimination than other quality reporting programs, protecting providers from being inaccurately classified as outliers, and at the same time identifying approximately 25 percent of surgical groups as different from average (either 3 stars or 1 star overall). Limitations/other issues To adjust for potential differences in the health of patients treated by each surgical group, risk scores are calculated for each surgical patient, and then averaged across all of a group s patients. The average risk is used in the risk adjustment, rather than individual patients risk scores. The latter would allow a more precise adjustment for individual health status, but also introduces enormous computational complexity. Consumers Union of U.S., Inc., 2011 6

The confidence limits for each provider are a function of that provider s sample size and the overall within- and between-provider variability. Consequently, providers with smaller numbers of procedures have wider confidence limits. This results in some providers with estimates further away from the mean failing to be significantly different from the mean, while other providers with larger numbers of patients may have less extreme means but are identified as being different from the mean. As with any statistical estimation process, all of these values have some inherent imprecision, and need to be interpreted accordingly. These ratings reflect the performance of surgical groups, not of individual surgeons. The ratings provide important information about a group s average performance, but a patient may want to ask more specific questions about surgeon-specific performance, which STS does not routinely calculate. Some of the data elements needed to calculate the measures used in this analysis can be difficult to define precisely, so it is possible that differences in data collection and coding can account for some of the differences among groups. Although the measures included in this analysis are endorsed by the National Quality Forum, cover both process and outcomes, and reflect pre-operative, operative, and postoperative care, there may be other important dimensions of surgical quality that are not included here. Examples include other important outcomes that are more elusive to measure, such as long-term recovery or quality of life. The STS is currently establishing linkages with other data sources to help determine long-term outcomes. Consumers Union of U.S., Inc., 2011 7

Appendix: Understanding the variation of specific ratings values The percentages calculated for each of the four domains for each practice are not precise values, but like all statistical measures, have some degree of uncertainty associated with them. For the statistical model used here, this margin of error is called a credible interval, analogous to the more traditional confidence interval. As a result, small differences between providers are not meaningful, and in many cases, it is not even meaningful to compare providers directly. As an example, consider the domain Absence of complications for surgical group A and surgical group B. The range and average of all surgical groups STS Min 48% STS Mean 86% STS Max 96% In the group with the lowest score, 48% of patients did not experience any of the major complications. On average, 86% of patients did not experience any of the major complications. In the group with the best score, 96% of patients did not experience any of the major complications. Group A s performance STS Min 48% The margin of error * for Group A extends this far STS Mean 86% 92% A STS Max 96% * 99% Bayesian credible interval 92% of patients in Group A did not experience any of the major complications Group B s performance STS Min 48% STS Mean 85% STS Max 96% 91% of patients in Group B did not experience any of the major complications, with this margin of error 91% B Group B has a wider margin of error because it has less data The margin of error for Group A is entirely above the mean, so Group A is better than average and earns 3 stars. The margin of error for Group B overlaps the mean, so Group B is average and earns 2 stars. Group A and Group B earn different star ratings, even though the difference between them is too small to be meaningful. Consumers Union of U.S., Inc., 2011 8